Impact of simulation and reference catalogues on the evaluation of taxonomic profiling pipelines

Abstract

Microbiome profiling tools rely on reference catalogues, which signifi-cantly affect their performance. Comparing them is however challenging, mainly due to differences in their native catalogues. In this study, we present a novel standardized benchmarking framework that makes such comparisons more accurate. Specifically, we conducted three realistic sim-ulations of gut microbiome samples, each based on a specific taxonomic profiler, and used two taxonomic references to project their results: the Genome Taxonomy Database and the Integrated Gene Catalogue. To demonstrate the importance of using such framework, we evaluated four established profilers along with a novel one, BiomScope. We evaluated the impact of these simulations and that of the common taxonomic references on the perceived performance of these profilers and provide guidelines.