Why Publishing Everything Is More Effective than Selective Publishing of Statistically Significant Results

Abstract

BackgroundDe Winter and Happee [1] examined whether science based on selective publishing of significant results may be effective in accurate estimation of population effects, and whether this is even more effective than a science in which all results are published (i.e., a science without publication bias). Based on their simulation study they concluded that “selective publishing yields a more accurate meta-analytic estimation of the true effect than publishing everything, (and that) publishing nonreplicable results while placing null results in the file drawer can be beneficial for the scientific collective” (p.4).Methods and FindingsUsing their scenario with a small to medium population effect size, we show that publishing everything is more effective for the scientific collective than selective publishing of significant results. Additionally, we examined a scenario with a null effect, which provides a more dramatic illustration of the superiority of publishing everything over selective publishing.ConclusionPublishing everything is more effective than only reporting significant outcomes.

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

“…Although the Proteus phenomenon may lead to unbiased effect size estimation, neglecting to publish studies with nonsignificant results is a very inefficient scientific enterprise with problems for statistical modeling of effect sizes (Van Assen, Van Aert, Nuijten, & Wicherts, 2014b, 2014c. Furthermore, even though there are occurrences of the Proteus phenomenon in some fields (Ioannidis, 2011), in psychology the vast majority of studies test if an effect is significantly different from zero, rather than if an effect is significantly different from a previously estimated effect (Fanelli, 2010(Fanelli, , 2012Van Assen, Van Aert, et al, 2014b).…”

“…Eliminating publication bias does not only increase accuracy of effect size estimates, it can also be shown that it is more efficient in terms of the number of studies that have to be conducted to estimate an effect with a certain level of precision (van Assen, van Aert, Nuijten, & Wicherts, 2014a).…”

“…Another way to decrease the prevalence of overestimated effects, is to eliminate publication bias altogether (van Assen, van Aert, et al, 2014a). It is widely recognized that publication bias is harmful, and many people have suggested potential solutions.…”

Can the behavioral sciences self-correct? A social epistemic study

“…The epistemic goal of the community is to discover that e↵ect. To do so, they perform experimental replications and cumulative meta-analyses (Cumming, 2012;de Winter & Happee, 2013;van Assen et al, 2014). The basic set-up is as follows.…”

Using synthetic biology to explore principles of development

“…but "That's funny…" '. Just as negative results in science are a potentially important but underreported resource (Sandercock, 2012;van Assen et al, 2014;Matsuda et al, 2015;Weintraub, 2016), 'engineering failures' in synthetic biology might become a valuable resource if more were published openly rather than being written off as mis-steps.…”