The impact of incomplete knowledge on the evaluation of protein function prediction: a structured-output learning perspective

Abstract

Motivation: The automated functional annotation of biological macromolecules is a problem of computational assignment of biological concepts or ontological terms to genes and gene products. A number of methods have been developed to computationally annotate genes using standardized nomenclature such as Gene Ontology (GO). However, questions remain about the possibility for development of accurate methods that can integrate disparate molecular data as well as about an unbiased evaluation of these methods. One important concern is that experimental annotations of proteins are incomplete. This raises questions as to whether and to what degree currently available data can be reliably used to train computational models and estimate their performance accuracy.Results: We study the effect of incomplete experimental annotations on the reliability of performance evaluation in protein function prediction. Using the structured-output learning framework, we provide theoretical analyses and carry out simulations to characterize the effect of growing experimental annotations on the correctness and stability of performance estimates corresponding to different types of methods. We then analyze real biological data by simulating the prediction, evaluation and subsequent re-evaluation (after additional experimental annotations become available) of GO term predictions. Our results agree with previous observations that incomplete and accumulating experimental annotations have the potential to significantly impact accuracy assessments. We find that their influence reflects a complex interplay between the prediction algorithm, performance metric and underlying ontology. However, using the available experimental data and under realistic assumptions, our results also suggest that current large-scale evaluations are meaningful and almost surprisingly reliable.Contact: predrag@indiana.eduSupplementary information: Supplementary data are available at Bioinformatics online.

Comparative transcriptome profiling of the human and mouse dorsal root ganglia: An RNA-seq-based resource for pain and sensory neuroscience research

“…Based on the Enrichr tool [48], we performed enrichment analysis for biological processes in the Gene Ontology (GO) database [3] (Table 6) and molecular pathways in the Reactome database [17] ( Table 7) that were enriched in our gene set (Fisher's exact test, p < 0.01 after multiple testing correction). Gene or protein functional ontologies are well known to be incomplete [40], necessitating us to look at DRG-enriched genes in non-enriched ontology terms and the literature, and showing the diverse roles played by these gene products. Several of these are already well studied in DRG biology.…”

The Ortholog Conjecture Revisited: the Value of Orthologs and Paralogs in Function Prediction

“…It is important to mention that all functional similarity measures between proteins are susceptible to problems caused by incomplete [12] and noisy [45] experimental annotations. There is a small effect of annotation incompleteness on topological measures and a somewhat larger effect on unnormalized semantic distance [27]. However, compared with topological measures, semantic similarity avoids a form of double-counting of nodes caused by the directed acyclic graph structure of GO, and thus properly treats hierarchical dependencies in the ontology.…”

Semantic Similarity in the Gene Ontology

“…This is due to the fact that annotations are not uniformly distributed among the proteins within an annotation corpus (and also vary among different organisms corpora), with some proteins being very well annotated while others have a single annotation. Both of these issues stem from incomplete annotations, which have been shown to have a signifi cant impact in the performance of information-theoretic measures [ 27 ]. Finally, SS approaches need to be aware of the impact that using electronic annotations (evidence code IEA) can have.…”