Is preregistration worthwhile?

Abstract

Proponents of preregistration argue that, among other benefits, it improves the diagnosticity of statistical tests [1]. In the strong version of this argument, preregistration does this by solving statistical problems, such as family-wise error rates. In the weak version, it nudges people to think more deeply about their theories, methods, and analyses. We argue against both: the diagnosticity of statistical tests depend entirely on how well statistical models map onto underlying theories, and so improving statistical techniques does little to improve theories when the mapping is weak. There is also little reason to expect that preregistration will spontaneously help researchers to develop better theories (and, hence, better methods and analyses).

Why diverse perspectives matter for Memory & Cognition

“…Rather, I firmly believe that multiple experiments are required to develop reliable methods and establish a predictable pattern of results. These values are consistent with the underlying principles driving the preregistration movement (e.g., Nosek, Ebersole, DeHaven, & Mellor, 2018), but like many others (e.g., Oberauer & Lewandowsky, 2019;Szollosi et al, 2019), I am confident that preregistration alone will not yield reliable research and strong theory. This is why a strong editorial team with a commitment to these ideals will allow M&C to act within its mission moving forward.…”

Lack of theory building and testing impedes progress in the factor and network literature

“…As Cummins put it: "In psychology, we are overwhelmed with things to explain, and somewhat underwhelmed by things to explain them with" (Cummins, 2000). For this reason, many scholars have argued that psychology's attention to statistics and replicability has distracted from a problem that runs much deeper: a crisis of theory (Borsboom, 2013;Borsboom et al, 2020;Cummins, 2000;Guest & Martin, 2020;Haslbeck et al, 2019;Meehl, 1990b;Muthukrishna & Henrich, 2019;Phaf, 2020;Robinaugh, Haslbeck, et al, 2019;Smaldino, 2019;Szollosi et al, 2019;Vaidyanathan et al, 2015;Van Rooij & Baggio, 2020).…”

“…Such work features explicit theories that provide more precise explanations and predictions, and allows for testing whether predictions are realized in data. Formalizing such theories in mathematical or computational models to escape vagaries of language is a critical step forward, and has been successful in many 33 other disciplines (Epstein, 2008;Lakens & Debruine, 2020;Morton, 2009;Oberauer & Lewandowsky, 2019;Robinaugh, Haslbeck, et al, 2019;Smaldino, 2017Smaldino, , 2019Szollosi et al, 2019).…”

“…Somewhat surprisingly, since starting to work on this paper in early 2019, numerous contributions in different areas of psychology have identified this crisis of theory as a crucial challenge moving forward (Borsboom et al, 2020;Burger et al, 2019;Gershman, 2019;Guest & Martin, 2020;Haslbeck et al, 2019;Kellen, 2019;Lakens & Debruine, 2020;Muthukrishna & Henrich, 2019;Oberauer & Lewandowsky, 2019;Robinaugh, Haslbeck, et al, 2019;Savi et al, 2019;Smaldino, 2019;Szollosi et al, 2019;Van Rooij & Baggio, 2020), and both the Journal of Abnormal Psychology and Perspective of Psychological Science have opened calls for special issues on theory.…”

Methodological Advances in Motor Learning and Development

“…Similarly, I think that pre-registration is an invaluable tool for increasing transparency (Nosek, Beck, Campbell, & Flake, 2019) and that adequate sample sizes are essential for replicable results (Button et al, 2013;Lohse, Buchanan, & Miller, 2016). However, it is also important that we do not mistake these practices as markers of improved scientific reasoning (Szollosi et al, 2019). To draw on a historical example, early physical investigations into the nature of temperature were highly replicable, but conceptualizing heat as a "caloric fluid" was still incorrect (Chang, 2004).…”