Over-optimism in unsupervised microbiome analysis: Insights from network learning and clustering

Abstract

In recent years, unsupervised analysis of microbiome data, such as microbial network analysis and clustering, has increased in popularity. Many new statistical and computational methods have been proposed for these tasks. This multiplicity of analysis strategies poses a challenge for researchers, who are often unsure which method(s) to use and might be tempted to try different methods on their dataset to look for the “best” ones. However, if only the best results are selectively reported, this may cause over-optimism: the “best” method is overly fitted to the specific dataset, and the results might be non-replicable on validation data. Such effects will ultimately hinder research progress. Yet so far, these topics have been given little attention in the context of unsupervised microbiome analysis. In our illustrative study, we aim to quantify over-optimism effects in this context. We model the approach of a hypothetical microbiome researcher who undertakes four unsupervised research tasks: clustering of bacterial genera, hub detection in microbial networks, differential microbial network analysis, and clustering of samples. While these tasks are unsupervised, the researcher might still have certain expectations as to what constitutes interesting results. We translate these expectations into concrete evaluation criteria that the hypothetical researcher might want to optimize. We then randomly split an exemplary dataset from the American Gut Project into discovery and validation sets multiple times. For each research task, multiple method combinations (e.g., methods for data normalization, network generation, and/or clustering) are tried on the discovery data, and the combination that yields the best result according to the evaluation criterion is chosen. While the hypothetical researcher might only report this result, we also apply the “best” method combination to the validation dataset. The results are then compared between discovery and validation data. In all four research tasks, there are notable over-optimism effects; the results on the validation data set are worse compared to the discovery data, averaged over multiple random splits into discovery/validation data. Our study thus highlights the importance of validation and replication in microbiome analysis to obtain reliable results and demonstrates that the issue of over-optimism goes beyond the context of statistical testing and fishing for significance.

Application of Computational Data Modeling to a Large-Scale Population Cohort Assists the Discovery of Inositol as a Strain-Specific Substrate for Faecalibacterium prausnitzii

“…Issues around over-optimism in microbiome analysis have recently been raised. Critiques on overfitting of data point out the potential pitfalls in reliability and reproducibility of the analysis [ 47 ]. In the current study, the performance of the model was stable with the AUC value of ROC being slightly higher in the test set than in the training set.…”

From RNA sequencing measurements to the final results: A practical guide to navigating the choices and uncertainties of gene set analysis

“…Such practice may seem natural at first glance so that many researchers who proceed this way may not be aware that this excessive fitting of the analysis strategy to the given data set is a form of cherry‐picking, also denoted as fishing for significance in the context of statistical tests. In other contexts than GSA, it has been demonstrated that cherry‐picking results in over‐optimistic research findings that cannot be validated on new, independent data (Ullmann et al, 2023) and thus contributes to the so‐called replication crisis. In this context, we want to emphasize that research findings that cannot be replicated on independent data are not valid.…”

Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices

“…This impression conflicts with a growing realization that there is a multiplicity of possible analysis strategies when analyzing empirical data [ 1 – 3 ] and that data analysts require the ability to make subjective decisions and acknowledge the multiplicity of possible perspectives [ 4 ]. In particular, so-called multianalyst projects [ 5 – 7 ] show that different teams of researchers make very different choices when they are asked to answer the same research question on the same data set.…”

“…This is a practice known as “p-hacking” or “fishing for significance” in the context of hypothesis testing and, more generally, “fishing expeditions” or “cherry-picking.” These practices lead to overconfident and nonreplicable research findings in the literature and, ultimately, to situations where some may argue that “most published research findings are false,” especially in combination with a low prior probability of the hypothesis being true [ 10 , 11 ]. Computational biology as a field is, unfortunately, not immune to these types of problems [ 3 , 12 ].…”

“…For example, Ullmann et al [ 3 ] show how the combination of researchers’ expectations and selective reporting may lead to overoptimistic results in the context of unsupervised microbiome analysis. Their paper highlights the relevance of open science practices in the field of computational biology.…”