Standardized benchmarking in the quest for orthologs

Abstract

Achieving high accuracy in orthology inference is essential for many comparative, evolutionary and functional genomic analyses, yet the true evolutionary history of genes is generally unknown and orthologs are used for very different applications across phyla, requiring different precision-recall trade-offs. As a result, it is difficult to assess the performance of orthology inference methods. here, we present a community effort to establish standards and an automated web-based service to facilitate orthology benchmarking. using this service, we characterize 15 well-established inference methods and resources on a battery of 20 different benchmarks. standardized benchmarking provides a way for users to identify the most effective methods for the problem at hand, sets a minimum requirement for new tools and resources, and guides the development of more accurate orthology inference methods.Evolutionarily related genes (homologs) across different species are often divided into gene pairs that originated through speciation events (orthologs) and gene pairs that originated through duplication events (paralogs) 1 . This distinction is useful in a broad range of contexts, including phylogenetic tree inference, genome annotation, comparative genomics and gene function prediction 2-4 . Accordingly, dozens of methods 5 and resources 6-8 for orthology inference have been developed.Because the true evolutionary history of genes is typically unknown, assessing the performance of these orthology inference methods is not straightforward. Several indirect approaches have been proposed. Based on the notion that orthologs tend to be functionally more similar than paralogs (a notion now referred to as the ortholog conjecture 9-12 ), Hulsen et al. 13 used several measures of functional conservation (coexpression levels, protein-protein interactions and protein domain conservation) to benchmark orthology inference methods. Chen et al. 14 proposed an unsupervised learning approach based on consensus among different orthology methods. Altenhoff and Dessimoz 15 introduced a phylogenetic benchmark measuring the concordance between gene trees reconstructed from putative orthologs and undisputed species trees. More recently, several 'gold standard' reference sets, either manually curated 16,17 or derived from trusted resources 18 , have been used as benchmarks. Finally, Dalquen et al. 19 used simulated genomes to assess orthology inference in the presence of varying amounts of duplication, lateral gene transfer and sequencing artifacts.This wide array of benchmarking approaches poses considerable challenges to developers and users of orthology methods. Conceptually, the choice of an appropriate benchmark strongly depends on the application at hand. Practically, most methods are not available as stand-alone programs and thus cannot easily be

Towards reproducible, transparent, and systematic benchmarking of omics computational tools

“…Access to data generated by benchmarking tools can easily improve the precision of newly developed tools by comparing a new method to tools indexed in the benchmarking study. Results of the benchmarking study can either be downloaded and the analysis locally run, or researchers can upload their own results and obtain a comparison through an online interface 17,18 . In both cases, benchmarking data allows researchers to easily compare newly developed tools against existing tools without installing and running third-party softwareoften an extremely complicated and time-consuming process, especially when the software lacks detailed documentation 17 .…”

“…al. 18 19 showed that the accuracy of cell population identification from flow cytometry data can be improved by combining predictions from individual computational algorithms.…”

“…List of names of the studies with corresponding citation used in Table 1. • A survey of error-correction methods for next-generation sequencing 40 • Critical assessment of automated flow cytometry data analysis techniques 19 • Assemblathon 2: evaluating de novo methods of genome assembly in three vertebrate species 41 • A comprehensive evaluation of assembly scaffolding tools 37 • An evaluation of the accuracy and speed of metagenome analysis tools 27,37 • Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software 8 • Comprehensive benchmarking and ensemble approaches for metagenomic classifiers 34 • A large-scale evaluation of computational protein function prediction 42 • An expanded evaluation of protein function prediction methods shows an improvement in accuracy 43 • Standardized benchmarking in the quest for orthologs 18 • Benchmarking short sequence mapping tools 44 • Alignathon: a competitive assessment of whole-genome alignment methods 45 • Simulation-based comprehensive benchmarking of RNA-seq aligners 6 • Characterization and improvement of RNA-Seq precision in quantitative transcript expression profiling 6,46 • Assessment of transcript reconstruction methods for RNA-seq 47 …”

Navigating the Phenotype Frontier: The Monarch Initiative

“…Sources differ as to whether given genes are orthologs and if so, what type (e.g., least diverged ortholog, paralog, one-to-one, many-to-many, etc.) (O'Brien et al 2005;Altenhoff et al 2016). In the above example, the zebrafish has two copies of the SKI ortholog: skia and skib.…”

Synteny-based phylogenomic networks for comparative genomics

“…The analysis of synteny in the gene order sense has several applications in genomics. Shared synteny is one of the most reliable criteria for establishing the orthology of genomic regions in different species (Dewey, 2011;Altenhoff et al, 2016). Additionally, exceptional conservation of synteny can reflect important functional relationships between genes.…”