The CAFA challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens

Abstract

The Critical Assessment of Functional Annotation (CAFA) is an ongoing, global, community-driven effort to evaluate and improve the computational annotation of protein function. Here we report on the results of the third CAFA challenge, CAFA3, that featured an expanded analysis over the previous CAFA rounds, both in terms of volume of data analyzed and the types of analysis performed. In a novel and major new development, computational predictions and assessment goals drove some of the experimental assays, resulting in new functional annotations for more than 1000 genes. Specifically, we performed experimental whole-genome mutation screening in Candida albicans and Pseudomonas aureginosa genomes, which provided us with genome-wide experimental data for genes associated with biofilm formation and motility (P. aureginosa only). We further performed targeted assays on selected genes in Drosophila melanogaster, which we suspected of being involved in long-term memory. We conclude that, while predictions of the molecular function and biological process annotations have slightly improved over time, those of the cellular component have not. Term-centric prediction of experimental annotations remains equally challenging; although the performance of the top methods is significantly better than expectations set by baseline methods in C. albicans and D. melanogaster, it leaves considerable room and need for improvement. We finally report that the CAFA community now involves a broad range of participants with expertise in bioinformatics, biological experimentation, biocuration, and bioontologies, working together to improve functional annotation, computational function prediction, and our ability to manage big data in the era of large experimental screens. 157 project. Predicting GO terms for a protein (protein-centric) and predicting which proteins are associated 158 with a given function (term-centric) are related but different computational problems: the former is a 159 multi-label classification problem with a structured output, while the latter is a binary classification task. 160Predicting the results of a genome-wide screen for a single or a small number of functions fits the term-centric 161 formulation. To see how well all participating CAFA methods perform term-centric predictions, we mapped 162 results from the protein-centric CAFA3 methods onto these terms. In addition we held a separate CAFA 163 challenge, CAFA-π whose purpose was to attract additional submissions from algorithms that specialize in 164 term-centric tasks. 165 We performed screens for three functions in three species, which we then used to assess protein function 166 prediction. In the bacterium Pseudomonas aeruginosa and the fungus Candida albicans we performed 167 genome-wide screens capable of uncovering genes with two functions, biofilm formation (GO:0042710) and 168 motility (for P. aeruginosa only) (GO:0001539), as described in Methods. In Drosophila melanogaster we 169 performed targeted assays, guided by previous CAFA submissions, of a ...

Hierarchical Markov Random Field model captures spatial dependency in gene expression, demonstrating regulation via the 3D genome

“…The function of gene products is closely correlated with its expression levels, since co-functioning genes are often co-transcribed [43]. The latest Critical Assessment of Functional Annotation (CAFA3) reported that the best computational model in predicting the proteins associated with a particular function takes advantage of expression data [44]. While PhiMRF serves as a statistical model to understand the relationship between expression and spatial organization, it would be interesting to see whether such relationship is particularly strong for certain functional groups of genes, which could potentially explain the regulation mechanism for such functions.…”

CATH functional families predict protein functional sites

“…FunFam relatives have been found to be more functionally similar than other domain-based resources (see Supplementary Section S1; . Function prediction pipelines based on FunFams have been consistently ranked among the best function prediction methods by the international CAFA competition (Jiang et al, 2016;Radivojac et al, 2013) and more recently have been amongst the top 5 best performing methods (CAFA3; (Zhou et al, 2019)).…”

Neural network and random forest models in protein function prediction

“…The official CAFA3 evaluation places our ensemble system in top-10 overall out of 68 93 participating teams and 144 submitted systems, with particularly strong performance 94 on molecular function and cellular component categories of prokaryotic proteins, where 95 the system placed 3rd and 2nd [27].…”

“…This is of course 338 a very difficult task. As shown in all CAFA challenges [1,2,27] the accuracy of the predictions varies greatly among the organisms and different ontologies. To look beyond 340 overall system performance, we next focus on the results of our methods on these two 341 aspects: ontologies and organisms.…”