Critical Assessment of Metagenome Interpretation - the second round of challenges

Abstract

Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the community-driven initiative for the Critical Assessment of Metagenome Interpretation (CAMI). In its second challenge, CAMI engaged the community to assess their methods on realistic and complex metagenomic datasets with long and short reads, created from ∼1,700 novel and known microbial genomes, as well as ∼600 novel plasmids and viruses. Altogether 5,002 results by 76 program versions were analyzed, representing a 22x increase in results.Substantial improvements were seen in metagenome assembly, some due to using long-read data. The presence of related strains still was challenging for assembly and genome binning, as was assembly quality for the latter. Taxon profilers demonstrated a marked maturation, with taxon profilers and binners excelling at higher bacterial taxonomic ranks, but underperforming for viruses and archaea. Assessment of clinical pathogen detection techniques revealed a need to improve reproducibility. Analysis of program runtimes and memory usage identified highly efficient programs, including some top performers with other metrics. The CAMI II results identify current challenges, but also guide researchers in selecting methods for specific analyses.

SemiBin: Incorporating information from reference genomes with semi-supervised deep learning leads to better metagenomic assembled genomes (MAGs)

“…For the benchmarking of binners, we used 5 simulated datasets from CAMI I and CAMI II and 5 real metagenomic datasets. Five simulated datasets of CAMI I and CAMI II were downloaded from CAMI challenge (Sczyrba et al, 2017; Meyer et al, 2021). CAMI I includes three datasets: low complexity, medium complexity, and high complexity datasets.…”

“…To compare the performance of SemiBin to existing binners, we first benchmarked SemiBin, Metabat2 (Kang et al, 2019), Maxbin2 (Wu et al, 2016), VAMB (Nissen et al), SolidBin (Wang et al, 2019), and COCACOLA (Lu et al, 2017) on the five simulated datasets from CAMI I and CAMI II (Sczyrba et al, 2017; Meyer et al, 2021) (the Critical Assessment of Metagenome Interpretation). The CAMI I datasets comprise different numbers of organisms including strain variation with low (40 genomes, 1 sample), medium (132 genomes, 2 samples), and high (596 genomes, 5 samples) complexity and were used to evaluate single-sample (low complexity) and co-assembly (medium and high complexity) binning.…”

A review of computational tools for generating metagenome-assembled genomes from metagenomic sequencing data

“…MetaBAT2 has been reported to demonstrate better performance than other binners, such as BMC3C, CONCOCT and MyCC, when using the CAMI I dataset [140] . However, despite having the quickest running time and lowest memory requirement, the performance of MetaBAT2 is worse than that of MaxBin2 and CONCOCT when using the CAMI II marine datasets, as indicated by a lower F1 score [141] . Notably, some ensemble strategy-based tools, such as MetaWRAP and DAS Tool, show excellent overall performance and generate high-quality scaffold clusters.…”

BugSplit: highly accurate taxonomic binning of metagenomic assemblies enables genome-resolved metagenomics

“…A community-initiative for the Critical Assessment of Metagenomics Interpretation (CAMI) has tracked the progress of this goal over time and posited two main challenges for metagenomic next-generation sequencing (mNGS): profiling and binning 2 . Metagenomic profiling aims to quantify the presence/absence and abundance of organisms in a microbial community, and has seen a marked improvement in the number and performance of tools from the first to the second CAMI challenge 3 .…”

“…While unsupervised binners have improved over recent years, their lack of ability to assign taxonomic labels to bins precludes downstream analyses such as profiling the presence/absence of specific organisms, performing organism-specific analyses, or identifying sequences of concern (eg. novel pathogens with sequence homology to known pathogens) 3 . The COVID-19 pandemic has highlighted the need for such improvements, with the aim of ensuring early availability of pathogen-agnostic diagnostics when outbreaks of novel strains emerge 5 .…”