An excess of positive results: Comparing the standard Psychology literature with Registered Reports

Abstract

When studies with positive results that support the tested hypotheses have a higher probability of being published than studies with negative results, the literature will give a distorted view of the evidence for scientific claims. Psychological scientists have been concerned about the degree of distortion in their literature due to publication bias and inflated Type-1 error rates. Registered Reports were developed with the goal to minimise such biases: In this new publication format, peer review and the decision to publish take place before the study results are known. We compared the results in the full population of published Registered Reports in Psychology (N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature (N = 152) by searching 633 journals for the phrase ‘test* the hypothes*’ (replicating a method by Fanelli, 2010). Analysing the first hypothesis reported in each paper, we found 96% positive results in standard reports, but only 44% positive results in Registered Reports. The difference remained nearly as large when direct replications were excluded from the analysis (96% vs 50% positive results). This large gap suggests that psychologists underreport negative results to an extent that threatens cumulative science. Although our study did not directly test the effectiveness of Registered Reports at reducing bias, these results show that the introduction of Registered Reports has led to a much larger proportion of negative results appearing in the published literature compared to standard reports.

Claims and statistical inference in animal physical cognition research.

“…We found that 62% of claims made in our sample of animal physical cognition papers were positive, 22% were negative and 16% were inconclusive. This figure of 62% positive claims is lower than previous reports of ~90% positive claims in psychology in general (Fanelli, 2010), and closer to the ~50% of positive findings from novel Registered Report studies in psychology (Scheel et al, 2020). This suggests that animal physical cognition research in general may not have an intense bias towards positive claims and positive results, and that animal physical cognition research is receptive to negative reports.…”

The past, present, and future of Registered Reports

“…[Insert Figure 6 here] Lack of standardisation. Previous analyses by Hardwicke et al 82 and Scheel et al 56 show that RRs are registered and reported inconsistently and, in many cases, even lack sufficient information to determine the specific hypotheses. This lack of specificity likely arises from the incompatibility between the 17 th -century traditional manuscript format -involving discursive and often vague prose documentationand the demand for quasi-mathematical precision within RRs.…”

The potential of functional near-infrared spectroscopy-based neurofeedback – a systematic review and recommendations for best practice

“…We note that adopting such methods is challenging: they involve additional costs such as more time in the preparation phase of the study and less flexibility (Allen and Mehler, 2019). However, recent preliminary meta-research suggests that the chances of publishing findings that do not meet traditional statistical thresholds increases remarkably for studies published as Registered Reports (Allen and Mehler, 2019;Scheel et al, 2020) while citation counts are comparable to traditional papers (Hummer et al, 2019). Hence, increased transparency not only benefits the field, but likely also individual authors.…”