Preregistration Is Neither Sufficient nor Necessary for Good Science

Abstract

To address widespread perceptions of a reproducibility crisis in the social sciences, a growing number of scholars recommend the systematic preregistration of empirical studies. The purpose of this article is to contribute to an epistemological dialogue on the value of preregistration in consumer research by identifying the limitations, drawbacks, and potential adverse effects of a preregistration system. After a brief review of some of the implementation challenges that commonly arise with preregistration, we raise three levels of issues with a system of preregistration. First, we identify its limitations as a means of advancing consumer knowledge, thus questioning the sufficiency of preregistration in promoting good consumer science. Second, we elaborate on why consumer science can progress even in the absence of preregistration, thereby also questioning the necessity of preregistration in promoting good consumer science. Third, we discuss serious potential adverse effects of preregistration, both at the individual researcher level and at the level of the field as a whole. We conclude by offering a broader perspective on the narrower role that preregistration can play within the general pursuit of building robust and useful knowledge about consumers.

Inflated false-positive risk in common regression analyses: A combinatorial analysis of model sets

“…However, there is an ongoing discussion about the effectiveness of this procedure (see e.g. Pham and Oh (2021) and J. Simmons et al (2021)).…”

“…It is, therefore, advisable that researchers plan and conduct their research differently; before collecting any data, they should preregister all variables, hypotheses, sample size justification, outlier criteria, and analyses they aim to run (Pham & Oh, 2021;J. Simmons, Nelson, & Simonsohn, 2021;van 't Veer & Giner-Sorolla, 2016).…”

Preregistration of studies with existing data

“…The most common criticism against preregistration is that it does not fit all types of studies (e.g., Pham and Oh, 2021). For example, in longitudinal studies, where the final data set is available only years after the beginning if a study.…”

The Benefits of Candidly Reporting Consumer Research

“…Third, especially in cases where it is easy and cheap to run and discard many studies, this approach may lead to effects that are highly replicable but are present only in very limited circumstances (i.e., a specific configuration of background variables). If initial exploratory studies are not preregistered, but are cheap and easy to run, authors could explore an effect using multiple sets of stimuli and a variety of contexts in order to calibrate a specific procedure that yields an effect (Pham & Tae Oh, 2021). The specific procedure could then be used to isolate mediators and moderators (i.e., generate hypotheses).…”

“…Encouraging exploratory studies has advantages. First, exploration is a valuable source of concepts and hypotheses (Alba, 2012;Lynch et al, 2012;Pham & Tae Oh, 2021). Exploration can be used to identify empirical regularities (Steinle, 1997).…”