Impact of simulation and reference catalogues on the evaluation of taxonomic profiling pipelines

Abstract

Microbiome profiling tools rely on reference catalogues, which significantly affect their performance. Comparing them is however challenging, mainly due to differences in their native catalogues. In this study, we present a novel standardized benchmarking framework that makes such comparisons more accurate. Specifically, we conducted three realistic simulations of gut microbiome samples, each based on a specific taxonomic profiler, and used two taxonomic references to project their results: the Genome Taxonomy Database and the Integrated Gene Catalogue. To demonstrate the importance of using such framework, we evaluated four established profilers along with a novel one, BiomScope. We evaluated the impact of these simulations and that of the common taxonomic references on the perceived performance of these profilers and provide guidelines.