Does preregistration improve the credibility of research findings?

Abstract

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing. This historical transparency is supposed to facilitate judgments about the credibility of the research findings. The present article provides a critical review of 17 of the reasons behind this argument. The article covers issues such as HARKing, multiple testing, p-hacking, forking paths, optional stopping, researchers' biases, selective reporting, test severity, publication bias, and replication rates. It is concluded that preregistration's historical transparency does not facilitate judgments about the credibility of research findings when researchers provide contemporary transparency in the form of (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.

A comment on Ellwood-Lowe et al. (2020/2021): Presentation and discussion of results should take the lead from the preregistration

“…Other paradigms are not free of such requirements either: one can hack a Bayes-factor and cherry-pick from machinelearned predictions. However, there are other ways of meeting these requirements and drawing robust inferences from data (see, for example, Giudice & Gangestad, 2021;Oberauer & Lewandowsky, 2019;Rubin, 2020). We would do well to both deeply understand how preregistration solves certain inferential problems, and also to move beyond it.…”

Improving evidence-based practice through preregistration of applied research: Barriers and recommendations

“…Preregistration is the practice of publicly (typically immediately but occasionally following an embargo period) publishing plans on central components of the research process (e.g., hypotheses, methods, and analyses) before data collection begins. Preregistration is typically endorsed based upon claims for a range of benefits for transparency and credibility (Rubin 2020), including differentiating between exploratory and confirmatory A range of activities that intentionally or unintentionally distort data in favor of a researcher's own hypotheses -or omissions in reporting such practices -including; selective inclusion of data, hypothesizing after the results are known (HARKing), and p-hacking.…”

“…Providing a more comprehensive and transparent account of the research process (not just what was completed but also the process) protects researchers from pressure to report positive outcomes (Olken 2015) and would provide a more constructive basis for extracting and synthesizing quality evidence for informing practice. As such, in the absence of higher standards of transparency, preregistration can hold value for improving the credibility of the research (Rubin 2020).…”

Good Scientific Practice in MEEG Research: Progress and Perspectives

“…For the scientific community in general, benefits are: 1) Increased transparency of experimental procedures, likely enhancing verifiability and reproducibility; 2) A reduction of suboptimal scientific practices (although the impact of pre-registration alone on such practices is debated (Devezer et al, 2020;Rubin, 2020)); and 3) Reduction of publication bias in the scientific literature. Interestingly, with respect to the last point, a comparison of registered versus conventional reports revealed a striking imbalance of 44% versus 96% of positive findings, respectively (Scheel et al, 2021).…”