Low statistical power in biomedical science: a review of three human research domains

Dumas-Mallet, Estelle; Button, Katherine S; Boraud, Thomas; Gonon, François; Munafò, Marcus R.

doi:10.1098/rsos.160254

Cited by 179 publications

(151 citation statements)

References 21 publications

Supporting

Mentioning

141

Contrasting

Order By: Relevance

“…ad iv) The prediction of a bimodal distribution of power is well corroborated by evidence [10,14,15]. Particularly the lack of a mode around 80% power in our model, as well as all empirical studies is notable.…”

Section: Model Predictions and Empirical Evidencesupporting

confidence: 83%

“…Similarly, a scientific optimality model assuming true and false positive publications have equal but opposite scientific value, suggests more unlikely findings merit larger power [29, see Fig5 therein]. All these considerations suggest high impact journals should contain larger sample sizes, highlighting a need for explanation.ad ii) The available evidence suggests a negative correlation between effect size and sample size, seemingly contradicting our prediction[15,17,31]. However, the authors caution in the interpretation of this result due to the winner's curse phenomenon[10,17,32].…”

mentioning

confidence: 70%

See 1 more Smart Citation

A simple model suggesting economically rational sample-size choice drives irreproducibility

Braganza

2020

PLoS ONE

View full text Add to dashboard Cite

Several systematic studies have suggested that a large fraction of published research is not reproducible. One probable reason for low reproducibility is insufficient sample size, resulting in low power and low positive predictive value. It has been suggested that insufficient sample-size choice is driven by a combination of scientific competition and 'positive publication bias'. Here we formalize this intuition in a simple model, in which scientists choose economically rational sample sizes, balancing the cost of experimentation with income from publication. Specifically, assuming that a scientist's income derives only from 'positive' findings (positive publication bias) and that individual samples cost a fixed amount, allows to leverage basic statistical formulas into an economic optimality prediction. We find that if effects have i) low base probability, ii) small effect size or iii) low grant income per publication, then the rational (economically optimal) sample size is small. Furthermore, for plausible distributions of these parameters we find a robust emergence of a bimodal distribution of obtained statistical power and low overall reproducibility rates, both matching empirical findings. Overall, the model describes a simple mechanism explaining both the prevalence and the persistence of small sample sizes, and is well suited for empirical validation. It suggests economic rationality, or economic pressures, as a principal driver of irreproducibility.

show abstract

Section: Model Predictions and Empirical Evidencesupporting

confidence: 83%

mentioning

confidence: 70%

A simple model suggesting economically rational sample-size choice drives irreproducibility

Braganza

2020

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Two major factors contribute to this problem, namely inflated false positive rates [6] and lack of statistical power, i.e. inflated false negative rates [7][8][9][10]. While the detrimental effect of false positives is widely recognized, it may be less well known that inflated false negative rates may also diminish reproducibility, see Fig.…”

Section: Introductionmentioning

confidence: 99%

Inflated false negative rates undermine reproducibility in task-based fMRI

Lohmann

Stelzer

Mueller

et al. 2017

Preprint

View full text Add to dashboard Cite

Reproducibility is generally regarded as a hallmark of scientific validity. It can be undermined by two very different factors, namely inflated false positive rates or inflated false negative rates. Here we investigate the role of the second factor, i.e. the degree to which true effects are not detected reliably. The availability of large public databases and also supercomputing allows us to tackle this problem quantitatively. Specifically, we estimated the reproducibility in task-based fMRI data over different samples randomly drawn from a large cohort of subjects obtained from the Human Connectome Project. We use the full cohort as a standard of reference to approximate true positive effects, and compute the fraction of those effects that was detected reliably using standard software packages at various smaller sample sizes. We found that with standard sample sizes this fraction was less than 25 percent. We conclude that inflated false negative rates are a major factor that undermine reproducibility. We introduce a new statistical inference algorithm based on a novel test statistic and show that it improves reproducibility without inflating false positive rates.

show abstract

“…The high FNR and high FDR could be among the most important explanations for low reproducibility in RS-fMRI studies of brain disorders. This concern has frequently been addressed in biomedical research [19][20][21][22][23][24][25][26] .…”

Section: Discussionmentioning

confidence: 99%

Small effect size leads to reproducibility failure in resting-state fMRI studies

Jia

Zhao

Barton

et al. 2018

Preprint

View full text Add to dashboard Cite

Thousands of papers using resting-state functional magnetic resonance imaging (RS-fMRI) have been published on brain disorders. Results in each paper may have survived correction for multiple comparison. However, since there have been no robust results from large scale meta-analysis, we do not know how many of published results are truly positives. The present meta-analytic work included 60 original studies, with 57 studies (4 datasets, 2266 participants) that used a between-group design and 3 studies (1 dataset, 107 participants) that employed a within-group design. To evaluate the effect size of brain disorders, a very large neuroimaging dataset ranging from neurological to psychiatric disorders together with healthy individuals have been analyzed. Parkinson's disease off levodopa (PD-off) included 687 participants from 15 studies. PD on levodopa (PD-on) included 261 participants from 9 studies. Autism spectrum disorder (ASD) included 958 participants from 27 studies. The meta-analyses of a metric named amplitude of low frequency fluctuation (ALFF) showed that the effect size (Hedges' g) was 0.19 -0.39 for the 4 datasets using between-group design and 0.46 for the dataset using within-group design. The effect size of PD-off, PD-on and ASD were 0.23, 0.39, and 0.19, respectively. Using the meta-analysis results as the robust results, the between-group design results of each study showed high false negative rates (median 99%), high false discovery rates (median 86%), and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. The findings were similar for 4 RS-fMRI metrics including ALFF, regional homogeneity, and degree centrality, as well as for another widely used RS-fMRI metric namely seed-based functional connectivity. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the effect sizes are relatively small. Meta-analysis on un-thresholded t-maps is critical for the recovery of ground truth. We recommend that to achieve high reproducibility through meta-analysis, the neuroimaging research field should share raw data or, at minimum, provide un-thresholded statistical images.

show abstract

Low statistical power in biomedical science: a review of three human research domains

Cited by 179 publications

References 21 publications

A simple model suggesting economically rational sample-size choice drives irreproducibility

A simple model suggesting economically rational sample-size choice drives irreproducibility

Inflated false negative rates undermine reproducibility in task-based fMRI

Small effect size leads to reproducibility failure in resting-state fMRI studies

Contact Info

Product

Resources

About