Crowdsourced benchmarking of taxonomic metagenome profilers: lessons learned from the sbv IMPROVER Microbiomics challenge

Abstract

Background Selection of optimal computational strategies for analyzing metagenomics data is a decisive step in determining the microbial composition of a sample, and this procedure is complex because of the numerous tools currently available. The aim of this research was to summarize the results of crowdsourced sbv IMPROVER Microbiomics Challenge designed to evaluate the performance of off-the-shelf metagenomics software as well as to investigate the robustness of these results by the extended post-challenge analysis. In total 21 off-the-shelf taxonomic metagenome profiling pipelines were benchmarked for their capacity to identify the microbiome composition at various taxon levels across 104 shotgun metagenomics datasets of bacterial genomes (representative of various microbiome samples) from public databases. Performance was determined by comparing predicted taxonomy profiles with the gold standard. Results Most taxonomic profilers performed homogeneously well at the phylum level but generated intermediate and heterogeneous scores at the genus and species levels, respectively. kmer-based pipelines using Kraken with and without Bracken or using CLARK-S performed best overall, but they exhibited lower precision than the two marker-gene-based methods MetaPhlAn and mOTU. Filtering out the 1% least abundance species—which were not reliably predicted—helped increase the performance of most profilers by increasing precision but at the cost of recall. However, the use of adaptive filtering thresholds determined from the sample’s Shannon index increased the performance of most kmer-based profilers while mitigating the tradeoff between precision and recall. Conclusions kmer-based metagenomic pipelines using Kraken/Bracken or CLARK-S performed most robustly across a large variety of microbiome datasets. Removing non-reliably predicted low-abundance species by using diversity-dependent adaptive filtering thresholds further enhanced the performance of these tools. This work demonstrates the applicability of computational pipelines for accurately determining taxonomic profiles in clinical and environmental contexts and exemplifies the power of crowdsourcing for unbiased evaluation.

SingleM and Sandpiper: Robust microbial taxonomic profiles from metagenomic data

“…The taxonomic profiling accuracy of SingleM was first benchmarked on simulated communities which contained genomes from known species, testing against other tools for which a GTDB R207 reference database was available. Complex microbial communities were modelled after the CAMI 2 'marine' benchmark datasets (Meyer et al 2022). We found the performance of SingleM was superior, at an average of >0.13 better Bray-Curtis dissimilarity than all other tools at the species level (Figure 2A).…”

“…Despite the wide variety of profiling tools that have been developed and extensively benchmarked, accurate estimation of community composition remains a challenging problem (Meyer et al 2022;Poussin et al 2022). Existing taxonomic profiling software is also largely restricted to characterising the abundance of species with reference genomes, missing most novel species.…”

Roadmap for the integration of gastro‐intestinal (GI) tract microbiomes (human and domestic animal) in risk assessments under EFSA's remit

“…Results are evaluated using the same metrics by the organizing committee. Several metagenomics community evaluations have been organised: the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI) which has celebrated two editions (Sczyrba et al, 2017;; and the IMPROVER Microbiomics Challenge (Poussin et al, 2022). None of them have been particularly focused on gut microbiome analysis, but represent general metagenomics challenges.…”

“…In the IMPROVER Microbiomics Challenge (Poussin et al al., 2022), benchmark datasets covered a broad set of features in terms of habitat, host origin, dataset complexity and sequencing technique. The participants were asked to predict the composition and relative abundance of bacterial communities (at the phylum, genus, and species levels).…”

“…In this sense, several initiatives in the form of open challenges have been created to evaluate the performance of metagenomics software, as well as to investigate the robustness of the corresponding results by extended post-challenge analyses. We have identified two of these activities, namely CAMI (the Initiative for the Critical Assessment of Metagenome Interpretation) (Sczyrba et al, 2017; and the IMPROVER Microbiomics Challenge (Poussin et al, 2022). The results of these benchmarks are also taking into account when making global rankings of the tools.…”