Artificial microbiome heterogeneity spurs six practical action themes and examples to increase study power-driven reproducibility

Abstract

With >70,000 yearly publications using mouse data, mouse models represent the best engrained research system to address numerous biological questions across all fields of science. Concerns of poor study and microbiome reproducibility also abound in the literature. Despite the well-known, negativeeffects of data clustering on interpretation and study power, it is unclear why scientists often house >4 mice/cage during experiments, instead of ≤2. We hypothesized that this high animal-cage-density practice abounds in published literature because more mice/cage could be perceived as a strategy to reduce housing costs. Among other sources of 'artificial' confounding, including cyclical oscillations of the 'dirty-cage/excrement microbiome', we ranked by priority the heterogeneity of modern husbandry practices/perceptions across three professional organizations that we surveyed in the USA. Data integration (scoping-reviews, professional-surveys, expert-opinion, and 'implementability-scorestatistics') identified Six-Actionable Recommendation Themes (SART) as a framework to re-launch emerging protocols and intuitive statistical strategies to use/increase study power. 'Cost-vs-science' discordance was a major aspect explaining heterogeneity, and scientists' reluctance to change. With a 'housing-density cost-calculator-simulator' and fully-annotated statistical examples/code, this themedframework streamlines the rapid analysis of cage-clustered-data and promotes the use of 'study-powerstatistics' to self-monitor the success/reproducibility of basic and translational research. Examples are provided to help scientists document analysis for study power-based sample size estimations using preclinical mouse data to support translational clinical trials, as requested in NIH/similar grants or publications.According to a U.S. National Science Foundation subcommittee on science replicability, "reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results" 1 . More recently, reproducibility as a scientific concept has been proposed to be divided into three types: methods reproducibility, results reproducibility, and inferential reproducibility. While these terms are applied predominantly to the biomedical field, they are not without utility across other scientific fields, each of which are governed by their own internalized needs and criteria for "proof " 2,3 . With 73,363 PubMed publications using 'mice' in 2018, laboratory mice represent a critical

Patterns of ‘Analytical Irreproducibility’ in Multimodal Diseases

“…One important caveat to consider across animal studies is that increasing N alone is futile if clustered-data statistics are not used to control for animal cage-density (>1 mouse/cage), which our group showed contributes to ‘artificial heterogeneity’, ‘cyclical microbiome bias’, and false-positive/false-negative conclusions 2,54 . To infer the role of scientific decision on the need for particular statistical methods, we examined the studies reviewed by Walter et al .…”

“…However, to differentiate the causal connection between microbiome alterations and human diseases (from that of secondary alterations due to disease), animal models, primarily germ-free rodents transplanted with human gut/fecal microbiota ( hGM-FMT ), have been critical as in vivo phenotyping tools for human diseases. Unfortunately, despite considerable efforts from organizations and guidelines to help scientists design and report preclinical experiments ( e.g ., ARRIVE) 1,2 , there are still concerns of study reproducibility.…”

“…Studies have described novel technical sources of ‘artificial’ microbiome heterogeneity that could explain why hGM-FMT study results vary 2–6 . In our own work 2 , we discovered that scientists lacked appropriate methods for the description and analysis of cage-clustered data.…”

“…Studies have described novel technical sources of ‘artificial’ microbiome heterogeneity that could explain why hGM-FMT study results vary 2–6 . In our own work 2 , we discovered that scientists lacked appropriate methods for the description and analysis of cage-clustered data. To help scientists to self-correct issues on rodent experimentation, we identified ‘six action themes’ and provided examples, and statistical code, on how to use and compute ‘study power’ as a reproducible parameter that could enable inter-laboratory comparisons and improve the planning of human clinical trials based on preclinical data 2 .…”

“…After examining the statistical content of 38 studies 8–45 listed in Walter et al , 7 we found that scientists who increased N, concurrently reduced the number of MxD, indicating that statistically, scientists replace the disease variance in mice by the disease variance in humans in their hGM-FMT studies. Further, studies lacked proper clustered-data statistics to control for animal density; which is a major source of misleading results (false-positive, or false-negative), especially when scientists prefer to house many rodents per cage, and when the number of mice per experiment is low 2,46 .…”