‘Statistical Irreproducibility’ Does Not Improve with Larger Sample Size: How to Quantify and Address Disease Data Multimodality in Human and Animal Research

Abstract

Poor study reproducibility is a concern in translational research. As a solution, it is recommended to increase sample size (N), i.e., add more subjects to experiments. The goal of this study was to examine/visualize data multimodality (data with >1 data peak/mode) as cause of study irreproducibility. To emulate the repetition of studies and random sampling of study subjects, we first used various simulation methods of random number generation based on preclinical published disease outcome data from human gut microbiota-transplantation rodent studies (e.g., intestinal inflammation and univariate/continuous). We first used unimodal distributions (one-mode, Gaussian, and binomial) to generate random numbers. We showed that increasing N does not reproducibly identify statistical differences when group comparisons are repeatedly simulated. We then used multimodal distributions (>1-modes and Markov chain Monte Carlo methods of random sampling) to simulate similar multimodal datasets A and B (t-test-p = 0.95; N = 100,000), and confirmed that increasing N does not improve the ‘reproducibility of statistical results or direction of the effects’. Data visualization with violin plots of categorical random data simulations with five-integer categories/five-groups illustrated how multimodality leads to irreproducibility. Re-analysis of data from a human clinical trial that used maltodextrin as dietary placebo illustrated multimodal responses between human groups, and after placebo consumption. In conclusion, increasing N does not necessarily ensure reproducible statistical findings across repeated simulations due to randomness and multimodality. Herein, we clarify how to quantify, visualize and address disease data multimodality in research. Data visualization could facilitate study designs focused on disease subtypes/modes to help understand person–person differences and personalized medicine.

Antigenic operon fragmentation and diversification mechanism in Bacteroidota impacts gut metagenomics and pathobionts in Crohn's disease microlesions

“…Stata functions were used to assess multimodality ( rfb- gene-cluster types observed vs. simulated random) as previously reported in our laboratory. 67 Together, data analysis represents over 636 analytical submissions made to the various bioinformatic resources described in this study.…”

The basis of antigenic operon fragmentation inBacteroidotaand commensalism

“…Data analysis was conducted using parametric and non-parametric statistics using parametric or non-parametric methods (e.g., student T-tests, ANOVAs) using the software STATA (v17), R, and GraphPad, which was used primarily to make bar plots or boxplot illustrations where significance is represented as asterisks based on the level of significance which was held at p<0.05(*), <0.001(**) and <0.0001(***). Stata functions were used to assess multimodality (minioperon types observed vs. simulated random) as previously reported in our laboratory (48). Together, data analysis represents over 636 analytical submissions made to the various bioinformatic resources described in this study.…”