The World of Research Has Gone Berserk: Modeling the Consequences of Requiring “Greater Statistical Stringency” for Scientific Publication

Abstract

In response to growing concern about the reliability and reproducibility of published science, researchers have proposed adopting measures of 'greater statistical stringency', including suggestions to require larger sample sizes and to lower the highly criticized 'p < 0.05' significance threshold. While pros and cons are vigorously debated, there has been little to no modeling of how adopting these measures might affect what type of science is published. In this paper, we develop a novel optimality model that, given current incentives to publish, predicts a researcher's most rational use of resources in terms of the number of studies to undertake, the statistical power to devote to each study, and the desirable pre-study odds to pursue. We then develop a methodology that allows one to estimate the reliability of published research by considering a distribution of preferred research strategies. Using this approach, we investigate the merits of adopting measures of 'greater statistical stringency' with the goal of informing the ongoing debate.

A simple model suggesting economically rational sample-size choice drives irreproducibility

“…Note, that alternative models, in which choice of b is endogenized, describe similar results [24,25].…”

“…Here, we describe a simple model in which sample-size choice is viewed as the result of competitive economic pressures rather than scientific deliberations (similar to [24,25]).…”

“…Recently, this economic argument has been formally explored in two related optimality models [24,25]. While similar to the present model in spirit and conclusion, these models contain some higher order parameters, creating challenges for empirical validation.…”

Inferring Models with Alternative Stable States from Independent Observations

“…2 The statistics demonstrated here constitute self-contained, baseline evidence that have 3 long anchored single-attractor theoretical or statistical models (A.M. Legendre, 1805), but are 4 only first steps in scientific inference (Wasserstein, Schirm, & Lazar, 2019). Practical 5 implications and novelty should also be considered as criteria for scientific research in addition 6 to traditional statistics such as P-value and confidence intervals (Campbell & Gustafson, 2019 9 2013). These theories also potentially score highly on novelty criteria such as pre-study 10 probability-the a priori probability that a study's null hypothesis is false, where a low 11 probability indicates novelty (Campbell & Gustafson, 2019 less curated research with independent replicates, pre-study probabilities may be low and thus 15 highly novel.…”

“…Practical 5 implications and novelty should also be considered as criteria for scientific research in addition 6 to traditional statistics such as P-value and confidence intervals (Campbell & Gustafson, 2019 9 2013). These theories also potentially score highly on novelty criteria such as pre-study 10 probability-the a priori probability that a study's null hypothesis is false, where a low 11 probability indicates novelty (Campbell & Gustafson, 2019 less curated research with independent replicates, pre-study probabilities may be low and thus 15 highly novel. So far, the combination of novelty and weak evidence has led to the periodic 16 waxing and waning of alternative stable states and multiple-attractor theories throughout the 17 history of ecology and other natural sciences (Monod, 1971;Thom, 1972;Scheffer et al, 2001).…”