Excess Significance Bias in the Literature on Brain Volume Abnormalities

Abstract

There are too many studies with statistically significant results in the literature on brain volume abnormalities. This pattern suggests strong biases in the literature, with selective outcome reporting and selective analyses reporting being possible explanations.

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

“…Although the Proteus phenomenon may lead to unbiased effect size estimation, neglecting to publish studies with nonsignificant results is a very inefficient scientific enterprise with problems for statistical modeling of effect sizes (Van Assen, Van Aert, Nuijten, & Wicherts, 2014b, 2014c. Furthermore, even though there are occurrences of the Proteus phenomenon in some fields (Ioannidis, 2011), in psychology the vast majority of studies test if an effect is significantly different from zero, rather than if an effect is significantly different from a previously estimated effect (Fanelli, 2010(Fanelli, , 2012Van Assen, Van Aert, et al, 2014b).…”

“…The results of meta-analyses inspire policies and treatments, so it is essential that the effects reported in them are valid. However, in many fields meta-analytic effects appear to be overestimated (Ferguson & Brannick, 2012;Ioannidis, 2011;Niemeyer, Musch, & Pietrowsky, 2012Sterne, Gavaghan, & Egger, 2000;Sutton, Duval, Tweedie, Abrams, & Jones, 2000). One of the main causes seems to be publication bias; the phenomenon that statistically significant findings have a higher probability of being published than nonsignificant findings (Greenwald, 1975).…”

“…An excess of significant findings has been established in many fields Button et al, 2013;Fanelli, 2012;Francis, 2014;Ioannidis, 2011;Kavvoura et al, 2008;Renkewitz, Fuchs, & Fiedler, 2011;Tsilidis, Papatheodorou, Evangelou, & Ioannidis, 2012). The 33 Estimated given a two independent samples comparison, assuming an effect size of d = .50 (based on estimates from meta-analyses) and a total sample size of 40, the median total sample size in psychology (Marszalek, Barber, Kohlhart, & Holmes, 2011).…”

“…For instance, there is evidence for publication bias in about 10% of the metaanalyses in the field of genetic associations (Ioannidis, 2011), in roughly 15% of the metaanalyses in psychotherapy (Niemeyer et al, 2012(Niemeyer et al, , 2013, in 20% to 40% of psychological metaanalyses (Ferguson & Brannick, 2012), in about 25%-50% of meta-analyses in the medical sciences (Sterne et al, 2000;Sutton et al, 2000), in 38%-50% of meta-analyses in ecology and evolution (Jennions & Moller, 2002), and in about 80% of meta-analyses in the field of communication sciences (Levine, Asada, & Carpenter, 2009). Although percentages of metaanalyses that are subject to publication bias do not seem to be impressively high, the power of publication bias tests was generally low in these meta-analyses.…”

Objective and subjective cognitive enhancing effects of mixed amphetamine salts in healthy people

“…These findings raise the question of why many published studies find large effects of amphetamine on cognitive performance with tests of memory, executive function and other cognitive processes. We believe that the answer is related to a set of problems, specifically low study power, flexibility in specific outcomes to be tested and publication bias against null results, which bedevil all branches of science, as explained in Ioannidis's (2005) provocatively titled article, -Why most published research findings are false.‖ The impact of these problems on psychology and neuroscience research, in the absence of any intentional malfeasance has been discussed by Ioannidis (2011), Lehrer (2010 and Simmons, Nelson & Simonsohn (2011) among others. Research on cognitive enhancement is not particularly susceptible to these problems, compared to other research topics, but neither is it immune to them.…”