The Preregistration Revolution

Abstract

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.methodology | open science | confirmatory analysis | exploratory analysis | preregistration P rogress in science is marked by reducing uncertainty about nature. Scientists generate models that may explain prior observations and predict future observations. Those models are approximations and simplifications of reality. Models are iteratively improved and replaced by reducing the amount of prediction error. As prediction error decreases, certainty about what will occur in the future increases. This view of research progress is captured by George Box's aphorism: "All models are wrong but some are useful" (1, 2).Scientists improve models by generating hypotheses based on existing observations and testing those hypotheses by obtaining new observations. These distinct modes of research are discussed by philosophers and methodologists as hypothesis-generating versus hypothesis-testing, the context of discovery versus the context of justification, data-independent versus data-contingent analysis, and exploratory versus confirmatory research (e.g., refs. 3-6). We use the more general terms--postdiction and prediction--to capture this important distinction.A common thread among epistemologies of science is that postdiction is characterized by the use of data to generate hypotheses about why something occurred, and prediction is characterized by the acquisition of data to test ideas about what will occur. In prediction, data are used to confront the possibility that the prediction is wrong. In postdiction, the data are already known and the postdiction is generated to explain why they occurred.Testing predictions is vital for establishing diagnostic evidence for explanatory claims. Testing predictions assesses the uncertainty of scientific models by observing how well the predictions account for new data. Generating postd...

Comparing Dream to Reality: An Assessment of Adherence of the First Generation of Preregistered Studies

“…One more implementation of research transparency is a so called blind analysis, in which all data analytic choices are made using a shuffled data set before doing the final analysis on the real data (MacCoun and Perlmutter, 2015) . Finally, researchers could choose to preregister their research, which involves freezing the analytic choices on a public third-party repository prior to seeing, or ideally prior to collecting, the data (Nosek, Ebersole, DeHaven, & Mellor, 2018). By specifying decisions before data collection, researcher degrees of freedom are restricted, and decisions that are made during the data collection and analysis cannot be mistakenly reported as a priori.…”

It's About Time: Answering the Call for Greater Precision in Research and Practice

“…Attempts to address these questions would afford opportunities to test more precise predictions regarding what underlies the transition from initiation to maintenance and provide potentially more valuable answers—the behavioural equivalent of predicting when and how much it will rain in April. Moreover, as Scholz () notes, efforts to pursue more precise predictions fit well within the infrastructure that has emerged to support strong scientific practices (Nosek, Ebersole, DeHaven, & Mellor, ; Shrout & Rodgers, ).…”

Software for designing rigorous and replicable preclinical research: The Experimental Design Accelerator

“…Finally, the software generates output to document all the design choices. This allows for an a priori design and data analysis plan that can be documented before the data are collected (Nosek, Ebersole, DeHaven, & Mellor, ). The design report could then be archived in publicly available, time‐stamped data bases, such as the Open Science Framework (https://osf.io).…”