Keeping up with the genomes: efficient learning of our increasing knowledge of the tree of life

Abstract

Background It is a computational challenge for current metagenomic classifiers to keep up with the pace of training data generated from genome sequencing projects, such as the exponentially-growing NCBI RefSeq bacterial genome database. When new reference sequences are added to training data, statically trained classifiers must be rerun on all data, resulting in a highly inefficient process. The rich literature of “incremental learning” addresses the need to update an existing classifier to accommodate new data without sacrificing much accuracy compared to retraining the classifier with all data. Results We demonstrate how classification improves over time by incrementally training a classifier on progressive RefSeq snapshots and testing it on: (a) all known current genomes (as a ground truth set) and (b) a real experimental metagenomic gut sample. We demonstrate that as a classifier model’s knowledge of genomes grows, classification accuracy increases. The proof-of-concept naïve Bayes implementation, when updated yearly, now runs in 1/4th of the non-incremental time with no accuracy loss. Conclusions It is evident that classification improves by having the most current knowledge at its disposal. Therefore, it is of utmost importance to make classifiers computationally tractable to keep up with the data deluge. The incremental learning classifier can be efficiently updated without the cost of reprocessing nor the access to the existing database and therefore save storage as well as computation resources.

Critical Assessment of Metagenome Interpretation: the second round of challenges

“…Bracken 47 and NBC++ (ref. 48 ) had completeness above 80% at either rank, and CCMetagen 49 , DUDes v.0.08 (ref. 50 ), LSHVec v.gsa 51 , Metalign 52 , MetaPalette 53 and MetaPhlAn v.cami1 more than 80% purity.…”

Bias at the third nucleotide of codon pairs in virus and host genomes

“…The exponentially increasing size of available sequence data creates computational challenges and this has led to the development of more efficient methods of comparing genomes 22 , 23 . Our comparison of third position bias in codon pairs adds a new tool for analysis of all types of genomes.…”

Quantitative Microbial Risk Assessment and Molecular Biology: Paths to Integration

“…Methods are in development to periodically update these that could prove important. 43 The second limitation with respect to the use of sequencing methods for quantification is that absolute abundances are needed for QMRA, rather than relative abundances. There are several potential routes to solve this problem, though at best they can be regarded as under development.…”

“…With all sequencing-based methods, the use of a robust and updated reference database is needed. Methods are in development to periodically update these that could prove important …”