A simple model suggesting economically rational sample-size choice drives irreproducibility

Abstract

Several systematic studies have suggested that a large fraction of published research is not reproducible. One probable reason for low reproducibility is insufficient sample size, resulting in low power and low positive predictive value. It has been suggested that insufficient sample-size choice is driven by a combination of scientific competition and 'positive publication bias'. Here we formalize this intuition in a simple model, in which scientists choose economically rational sample sizes, balancing the cost of experimentation with income from publication. Specifically, assuming that a scientist's income derives only from 'positive' findings (positive publication bias) and that individual samples cost a fixed amount, allows to leverage basic statistical formulas into an economic optimality prediction. We find that if effects have i) low base probability, ii) small effect size or iii) low grant income per publication, then the rational (economically optimal) sample size is small. Furthermore, for plausible distributions of these parameters we find a robust emergence of a bimodal distribution of obtained statistical power and low overall reproducibility rates, both matching empirical findings. Overall, the model describes a simple mechanism explaining both the prevalence and the persistence of small sample sizes, and is well suited for empirical validation. It suggests economic rationality, or economic pressures, as a principal driver of irreproducibility.

Proxyeconomics, a theory and model of proxy-based competition and cultural evolution

“…While we would maintain it is important to complement such research with approaches yielding quantitatively falsifiable predictions (e.g. [ 49 ]), this does suggest fundamental limits to quantitative approaches with respect to the problems at hand. More specifically, it suggests an indispensable role of qualitative research in the ongoing assessment of real societal systems.…”

“…Specifically, we propose to systematically investigate empirical idiosyncrasies of proxy-generation in order to predict meso-level patterns of corruption. Such predictions would allow the diagnosis of corruption within societal systems, as well as the design of mitigation strategies [ 1 , 13 , 49 , 151 ]. For instance, we have recently modelled how an informational idiosyncrasy of the academic publication process, namely positive publication bias, predicts specific ‘corrupt’ patterns of sample-size choices.…”

“…null effects). Under plausible assumptions, this idiosyncrasy should give rise to a bimodal distribution of achieved statistical power, and relatively low overall replicability, even though both would undermine the actual goal of academic competition [ 49 ]. Yet both have been found empirically, supporting the notion of excessive proxy orientation [ 152 – 154 ].…”

“…Yet both have been found empirically, supporting the notion of excessive proxy orientation [ 152 – 154 ]. Having identified not only a ‘corrupt pattern’, but also the informational idiosyncrasy at its source, now additionally allows the design of targeted mitigation strategies [ 49 , 155 ].…”

“…the proxy). It allows empirically verifiable predictions about potential differences between proxy and goal ([ 49 ], see §4.7). Similarly, quarterly earnings reports of a company will lead to a predictable focus on outcomes realizing within this specific time frame.…”