Information-theoretic evaluation of predicted ontological annotations

Predicting protein function has been a major goal of bioinformatics for several decades, and it has gained fresh momentum thanks to recent community-wide blind tests aimed at benchmarking available tools on a genomic scale. Sequence-based predictors, especially those performing homology-based transfers, remain the most popular but increasing understanding of their limitations has stimulated the development of complementary approaches, which mostly exploit machine learning. Here we present FFPred 3, which is intended for assigning Gene Ontology terms to human protein chains, when homology with characterized proteins can provide little aid. Predictions are made by scanning the input sequences against an array of Support Vector Machines (SVMs), each examining the relationship between protein function and biophysical attributes describing secondary structure, transmembrane helices, intrinsically disordered regions, signal peptides and other motifs. This update features a larger SVM library that extends its coverage to the cellular component sub-ontology for the first time, prompted by the establishment of a dedicated evaluation category within the Critical Assessment of Functional Annotation. The effectiveness of this approach is demonstrated through benchmarking experiments, and its usefulness is illustrated by analysing the potential functional consequences of alternative splicing in human and their relationship to patterns of biological features.

show abstract

“…The information content of a GO term t was estimated in a Bayesian framework as proposed by Clark and Radivojac36 using the equation…”

Section: Methodsmentioning

confidence: 99%

FFPred 3: feature-based function prediction for all Gene Ontology domains

Cozzetto

Minneci

Currant

et al. 2016

Sci Rep

110

View full text Add to dashboard Cite

show abstract

“…In particular, we measure the total amount of information accretion (IA; Methods; Clark and Radivojac, 2013) that was contributed by different predictors. We estimate that the E. coli genome has on average 29.2 bits/gene of currently known functional annotations spanning all three GO domains (Fig.…”

Section: The Present and Future Potential In Function Prediction Methodsmentioning

confidence: 99%

Extensive complementarity between gene function prediction methods

2016

View full text Add to dashboard Cite

Motivation: The number of sequenced genomes rises steadily, but we still lack the knowledge about the biological roles of many genes. Automatic function prediction (AFP) is thus a necessity. We hypothesize that AFP approaches which draw on distinct genome features may be useful for predicting different types of gene functions, motivating a systematic analysis of the benefits gained by obtaining and integrating such predictions. Results: Our pipeline amalgamates 5,133,543 genes from 2,071 genomes in a single massive analysis that evaluates five established genomic AFP methodologies. While 1,227 Gene Ontology terms yielded reliable predictions, the majority of these functions were accessible to only one or two of the methods. Moreover, different methods tend to assign a GO term to non-overlapping sets of genes. Thus, inferences made by diverse AFP methods display a striking complementary, both gene-wise and function-wise. Because of this, a viable integration strategy is to simply rely on a single mostconfident prediction per gene/function, instead of enforcing agreement across multiple AFP methods. Using an information-theoretic approach, we estimate that current databases contain 29.2 bits/gene of known E. coli gene functions. This can be increased by up to 5.5 bits/gene using individual AFP methods, or by 11 additional bits/gene upon integration, thereby providing a highly-ranking predictor on the CAFA2 community benchmark. Availability of more sequenced genomes boosts the predictive accuracy of AFP approaches and also the benefit from integrating them.

show abstract

“…Semantic distance is based on an assumption that the prior probability of a protein’s experimental annotation can be modeled by a Bayesian network in which the conditional probability tables are calculated from data (Clark and Radivojac, 2013). Here, we calculate misinformation ( mi ) and remaining uncertainty ( ru ) as

m i (P, T) = \sum_{v \in P - T} i a (v) and r u (P, T) = \sum_{v \in T - P} i a (v),

where

v \in V

is a vertex in the graph,

P (v)

is a set of its parents,

\Pr (v | P (v))

is the probability that a protein is experimentally annotated by v given that all its parents are a part of the annotation, and

i a (v) = - \log true(\Pr (v | P (v) true)

is information accretion.…”

Section: Methodsmentioning

confidence: 99%

The impact of incomplete knowledge on the evaluation of protein function prediction: a structured-output learning perspective

et al. 2014

Self Cite

View full text Add to dashboard Cite

Motivation: The automated functional annotation of biological macromolecules is a problem of computational assignment of biological concepts or ontological terms to genes and gene products. A number of methods have been developed to computationally annotate genes using standardized nomenclature such as Gene Ontology (GO). However, questions remain about the possibility for development of accurate methods that can integrate disparate molecular data as well as about an unbiased evaluation of these methods. One important concern is that experimental annotations of proteins are incomplete. This raises questions as to whether and to what degree currently available data can be reliably used to train computational models and estimate their performance accuracy.Results: We study the effect of incomplete experimental annotations on the reliability of performance evaluation in protein function prediction. Using the structured-output learning framework, we provide theoretical analyses and carry out simulations to characterize the effect of growing experimental annotations on the correctness and stability of performance estimates corresponding to different types of methods. We then analyze real biological data by simulating the prediction, evaluation and subsequent re-evaluation (after additional experimental annotations become available) of GO term predictions. Our results agree with previous observations that incomplete and accumulating experimental annotations have the potential to significantly impact accuracy assessments. We find that their influence reflects a complex interplay between the prediction algorithm, performance metric and underlying ontology. However, using the available experimental data and under realistic assumptions, our results also suggest that current large-scale evaluations are meaningful and almost surprisingly reliable.Contact: predrag@indiana.eduSupplementary information: Supplementary data are available at Bioinformatics online.

show abstract

Information-theoretic evaluation of predicted ontological annotations

Cited by 121 publications

References 17 publications

FFPred 3: feature-based function prediction for all Gene Ontology domains

FFPred 3: feature-based function prediction for all Gene Ontology domains

Extensive complementarity between gene function prediction methods

The impact of incomplete knowledge on the evaluation of protein function prediction: a structured-output learning perspective

Contact Info

Product

Resources

About