Assessment of statistical methods from single cell, bulk RNA-seq, and metagenomics applied to microbiome data

Abstract

Background: The correct identification of differentially abundant microbial taxa between experimental conditions is a methodological and computational challenge. Recent work has produced methods to deal with the high sparsity and compositionality characteristic of microbiome data, but independent benchmarks comparing these to alternatives developed for RNA-seq data analysis are lacking. Results: We compare methods developed for single-cell and bulk RNA-seq, and specifically for microbiome data, in terms of suitability of distributional assumptions, ability to control false discoveries, concordance, power, and correct identification of differentially abundant genera. We benchmark these methods using 100 manually curated datasets from 16S and whole metagenome shotgun sequencing. Conclusions: The multivariate and compositional methods developed specifically for microbiome analysis did not outperform univariate methods developed for differential expression analysis of RNA-seq data. We recommend a careful exploratory data analysis prior to application of any inferential model and we present a framework to help scientists make an informed choice of analysis methods in a dataset-specific manner.

Multivariable Association Discovery in Population-scale Meta-omics Studies

“…In particular, count-based models, due to their strong parametric assumptions on the distributions or parametric specifications of the mean-variance dependency, tend to have inflated FDR when the assumptions are violated. In sharp contrast to previous claims, however, compositionality-corrected methods such as ANCOM 14,22 as well as specialized normalization and transformation methods such as CLR 46 did not improve performance over non-compositional approaches 8,47 , consistent with recent findings that compositional methods may not always outperform non-compositional methods 32 . Importantly, these conclusions hold regardless of the nature of the modeling paradigm (i.e.…”

“…Notably, previous benchmarking in this area has only focused on differential abundance testing without the simultaneous consideration of multiple covariates and repeated measures 7-9 . Additionally, with the exception of Hawinkel et al 7 , these benchmarking efforts lacked important considerations to the extent that they either (i) did not consider FDR as a metric of evaluation 9,31,32 or (ii) misreported false positive rate as FDR 8 ( Methods ). While a majority of these studies made a final recommendation based on the traditional AUC metric or a combination of sensitivity and specificity, we argue that without considering the FDR-controlling behavior of a method, the AUC values alone are too optimistic to draw any meaningful conclusions about its practical utility.…”

“…Unlike other methods, ANCOM does not report p-values but instead returns logical indicators of whether a feature is differentially abundant based on a test statistic W. It is to be noted that in the presence of multiple covariates, ANCOM does not return statistically significant feature-metadata pairs with respect to every covariate in the model, making it infeasible for our multivariable setting. Also, we did not test the ANCOM method in repeated measures settings as it was too slow and unstable for assessment, as noted elsewhere 32 .…”

Progressive Shifts in the Gut Microbiome Reflect Prediabetes and Diabetes Development in a Treatment-Naive Mexican Cohort

“…This required careful modeling of the sequencing counts which often do not follow normal distributions. Here we chose negative binomial models as they model the high prevalence of zero read counts and have been shown to represent amplicon sequencing data well ( 42 ). Consequently, associations between the microbiome and clinical variables were identified by a robust normalization and testing strategy based on DESeq2 (see Materials and Methods).…”

Microbial Reference Frames Reveal Distinct Shifts in the Skin Microbiota after Cleansing

“…Using "reference frames", while still a new and novel approach to evaluating microbial associations, represents an advancement in techniques that shows promise in handling compositional data and may improve the conclusions we draw. While the "reference frame" approach does not designate p-value associations for differential abundance, it shows "greater ability to identify correct taxa" [38,39]. Therefore, it was interesting to apply Songbird to a low-biomass dataset such as the skin microbiota as the test case in the original article had explored a similar low-biomass environment [23].…”