The preregistration revolution

Abstract

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.methodology | open science | confirmatory analysis | exploratory analysis | preregistration P rogress in science is marked by reducing uncertainty about nature. Scientists generate models that may explain prior observations and predict future observations. Those models are approximations and simplifications of reality. Models are iteratively improved and replaced by reducing the amount of prediction error. As prediction error decreases, certainty about what will occur in the future increases. This view of research progress is captured by George Box's aphorism: "All models are wrong but some are useful" (1, 2).Scientists improve models by generating hypotheses based on existing observations and testing those hypotheses by obtaining new observations. These distinct modes of research are discussed by philosophers and methodologists as hypothesis-generating versus hypothesis-testing, the context of discovery versus the context of justification, data-independent versus data-contingent analysis, and exploratory versus confirmatory research (e.g., refs. 3-6). We use the more general terms--postdiction and prediction--to capture this important distinction.A common thread among epistemologies of science is that postdiction is characterized by the use of data to generate hypotheses about why something occurred, and prediction is characterized by the acquisition of data to test ideas about what will occur. In prediction, data are used to confront the possibility that the prediction is wrong. In postdiction, the data are already known and the postdiction is generated to explain why they occurred.Testing predictions is vital for establishing diagnostic evidence for explanatory claims. Testing predictions assesses the uncertainty of scientific models by observing how well the predictions account for new data. Generating postd...

Lessons from Pinocchio: Cues to deception may be highly exaggerated

“…Preregistration of coding and analyses would also mitigate this issue by reducing researchers' ability to select cues that might favor their hypotheses and their ability to change their hypotheses to suit the data (Kerr, 1998;Nosek et al, 2018). If they are not already, deception researchers ought to become familiar with the myriad ways they can inadvertently arrive at mistaken conclusions.…”

The possibility and desirability of replication in the humanities

“…Lessons learned in other disciplinary fields that make research more replicable should be taken into account as well (Bouter, 2018;Peels and Bouter, 2018). The most important measures to introduce in the humanities may be preregistration of studies and uploading detailed methods, data analysis plans and data sets to suitable portals (Nosek et al, 2018). Additionally, the development and use of reporting guidelines for study protocols, publications and data sets will most likely also be important for the humanities.…”

An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach

“…Gelman and Loken (2013and Loken ( , 2014 suggested preregistration as a potential solution to the forking paths problem (e.g., Nosek, Ebersole, DeHaven, & Mellor, 2017;Nosek & Lakens, 2014;Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012). However, this potential solution suffers from a major drawback.…”

“…Most recently, Nosek et al (2017) summarised four strategies that are intended to address samplecontingent analyses in the preregistration paradigm. The first two strategies blind researchers to the substantive research results.…”