Power failure: why small sample size undermines the reliability of neuroscience

Abstract

It has been claimed and demonstrated that many (and possibly most) of the conclusions drawn from biomedi-cal research are probably false 1. A central cause for this important problem is that researchers must publish in order to succeed, and publishing is a highly competitive enterprise, with certain kinds of findings more likely to be published than others. Research that produces novel results, statistically significant results (that is, typically p < 0.05) and seemingly 'clean' results is more likely to be published 2,3. As a consequence, researchers have strong incentives to engage in research practices that make their findings publishable quickly, even if those practices reduce the likelihood that the findings reflect a true (that is, non-null) effect 4. Such practices include using flexible study designs and flexible statistical analyses and running small studies with low statistical power 1,5. A simulation of genetic association studies showed that a typical dataset would generate at least one false positive result almost 97% of the time 6 , and two efforts to replicate promising findings in biomedicine reveal replication rates of 25% or less 7,8. Given that these publishing biases are pervasive across scientific practice, it is possible that false positives heavily contaminate the neuroscience literature as well, and this problem may affect at least as much, if not even more so, the most prominent journals 9,10. Here, we focus on one major aspect of the problem: low statistical power. The relationship between study power and the veracity of the resulting finding is under-appreciated. Low statistical power (because of low sample size of studies, small effects or both) negatively affects the likelihood that a nominally statistically significant finding actually reflects a true effect. We discuss the problems that arise when low-powered research designs are pervasive. In general, these problems can be divided into two categories. The first concerns problems that are mathematically expected to arise even if the research conducted is otherwise perfect: in other words, when there are no biases that tend to create statistically significant (that is, 'positive') results that are spurious. The second category concerns problems that reflect biases that tend to co-occur with studies of low power or that become worse in small, underpowered studies. We next empirically show that statistical power is typically low in the field of neuroscience by using evidence from a range of subfields within the neuroscience literature. We illustrate that low statistical power is an endemic problem in neuroscience and discuss the implications of this for interpreting the results of individual studies. Low power in the absence of other biases Three main problems contribute to producing unreliable findings in studies with low power, even when all other research practices are ideal. They are: the low probability of finding true effects; the low positive predictive value (PPV; see BOX 1 for definitions of key statistical terms) when an eff...

Post-traumatic stress disorder moderates the relationship between trauma exposure and chronic pain

“…The limited sample size increased the risk of capitalizing on random fluctuations and the risk of both false positive and false negative findings (Button et al, 2013) as well as leaving some combinations of PTSD diagnosis and demographic background variables rare, such as men with PTSD. The study was introduced as a study on traumatic stress, which may have led to more persons with personal experiences of trauma exposure volunteering to participate.…”

Not all that glitters is gold: A guide to critical appraisal of animal drug trials in epilepsy

“…It is recommended that the endpoints and the original statistical plan are prespecified 47, 48. All analyses should be reported.…”

“…Nonsignificant or negative results should also be reported to prevent other research groups from pursuing the same experiment. In the final results, it is recommended that direction and magnitude of the effect size, the statistical test and actual p‐value, and the 95% confidence intervals are reported 47, 48, 49…”

Tourette syndrome research highlights from 2016

“…Nevertheless, as the title of the 2016 Frontiers Research Topic suggests, TS research is beginning to grow from case series and small pilot studies into finding “new avenues through large-scale collaborative projects.” This change is important, especially in light of increasing recognition that reproducibility is an important concern in all of science, and clearly in neuroscience 95, 96 . Larger samples are important to addressing this concern, and give greater hope for important new discoveries in the future.…”