What type of Type I error? Contrasting the Neyman–Pearson and Fisherian approaches in the context of exact and direct replications

Abstract

The replication crisis has caused researchers to distinguish between exact replications, which duplicate all aspects of a study that could potentially affect the results, and direct replications, which duplicate only those aspects of the study that are thought to be theoretically essential to reproduce the original effect. The replication crisis has also prompted researchers to think more carefully about the possibility of making Type I errors when rejecting null hypotheses. In this context, the present article considers the utility of two types of Type I error probability: the Neyman-Pearson long run Type I error rate and the Fisherian sample-specific Type I error probability. It is argued that the Neyman-Pearson Type I error rate is inapplicable in social science because it refers to a long run of exact replications, and social science deals with irreversible units (people, social groups, and social systems) that make exact replications impossible. Instead, the Fisherian sample-specific Type I error probability is recommended as a more meaningful way to conceptualize false positive results in social science because it can be applied to each sample-specific decision about rejecting the same substantive null hypothesis in a series of direct replications. It is concluded that the replication crisis may be partly (not wholly) due to researchers' unrealistic expectations about replicability based on their consideration of the Neyman-Pearson Type I error rate across a long run of exact replications.

Replication and the Establishment of Scientific Truth

“…Statistical decisions are not error-free either. It has been suggested that some replication failures may be due to applications of the wrong type of Type I error testing procedure: Neyman-Pearson vs. Fisherian Type I error rate (Rubin, 2019). New statistics are no panacea either.…”

“…In fact, a so-called exact replication is just one type of replication or, at best, an approximation of the original study, and more generally, it is just another type of empirical study. "There are no critical tests of theories, and there are no objectively decisive replications" (Earp and Trafimow, 2015); no such thing as an exact or "direct" replication exists (Stroebe and Strack, 2014;Anderson et al, 2016;Rubin, 2019). Attempted exact replications cannot therefore become the final arbiters of truth any more than the original studies.…”

Does preregistration improve the credibility of research findings?

“…It is also important to recognize that low replication rates have been attributed to more than just questionable research practices. For example, low replication rates have been attributed to (a) insufficiently stringent evidence thresholds (Benjamin et al, 2018), (b) insufficiently lenient evidence thresholds (Devezer et al, 2020), (c) poor measurement (e.g., Loken & Gelman, 2017), (d) model misspecification (Devezer et al, 2020), (e) low power (e.g., Rossi, 1990), (f) poor theory (e.g., Oberauer & Lewandowsky, 2019;Szollosi & Donkin, 2019), (g) an underappreciation of the influence of hidden moderators (Rubin, 2019b), and (h) errors in substantive inference (Jussim et al, 2016;Rubin, 2017b, p. 274). Hence, it is unclear whether preregistration is targeting the right set of issues to increase replication rates.…”

“Repeated sampling from the same population?” A critique of Neyman and Pearson’s responses to Fisher

“…Hence, Neyman did not require researchers to use exactly the same testing conditions; only testing conditions that appear to be the same. This point is reassuring, because exact replications are impossible in a universe that is constantly and irreversibly changing (Nosek & Errington, 2020;Rubin, 2019;Stroebe & Strack, 2014;Zwaan et al, 2018).…”

“…Two types of non-exact replication have been distinguished: direct (sometimes called close) and conceptual (Rubin, 2019;Stroebe & Strack, 2014;Zwaan et al, 2018). In both cases, researchers assume that they have repeated the testing conditions that are equivalent to those of their original test, even if those conditions are not exactly the same as those of their original test.…”