The Regression Trap and Other Pitfalls of Replication Science—Illustrated by the Report of the Open Science Collaboration

What Can We Learn from Many Labs Replications?

“…The statistical problem is that there "is no single standard for evaluating replication success" (OSC, 4716-2). Because much has been written on this (e.g., Fiedler & Prager, 2018;Trafimow, 2018), I limit my discussion to the effect of regression shrinkage, because it invalidates an apparently straightforward and intuitively appealing criterion for evaluating replications: Replications are considered successful if they show a statistically significant effect in the same direction as the original study. As Trafimow (2018) has recently pointed out, this is a poor decision criterion.…”

“…Again, this difference can be attributed to regression toward the mean. Strong original effects are likely to shrink when replicated (Fiedler & Prager, 2018). However, because-unlike with p values-there is no (direct) pressure to publish only studies with large effect sizes, regression toward the mean can also, in cases of studies with relatively small effects, result in replications to have larger effects than the original study (Fiedler & Prager, 2018).…”

“…Even a more modest title such as "Estimating the Reproducibility of Psychological Research Published in 2008 in Three Prime Journals" might be somewhat of an overstatement. As Fiedler and Prager (2018) criticized, the instruction to select the last study in each article may have captured studies testing problematic and tangential hypotheses that were forced on researchers by reviewers.…”

What should comparative cognition expect from direct replication studies?

“…For example, if exactly estimating the "true" effect size would have yielded a p-value of .06 in a design, then only samples overestimating the effect size would be published. Hence, in direction replication studies, the replication p value would regress to the mean (.06), and the original study would have less than a 50% chance of a positive direct replication with the same sample size (Fiedler & Prager, 2018;Sterling, 1959). Second, the overestimation of published effect sizes is further exacerbated if research practices that produce more significant results are used, which appears relatively common (Fraser, Parker, Nakagawa, Barnett, & Fidler, 2018;John, Loewenstein, & Prelec, 2012;Simmons, Nelson, & Simonsohn, 2011).…”

Context Dependency as a Predictor of Replicability

“…the difficulty to replicate an original effect is that it never really existed in the first place (i.e., a "false positive"), and that its original statistical significance was artificially amplified ("phacking; " Bakker, van Dijk, & Wicherts, 2012;Simmons, Nelson, & Simonsohn, 2011) or the result of pure chance (an argument also fueled by a low base-rate of true effects; see Miller, 2009;Miller & Ulrich, 2016;Wilson & Wixted, 2018). A second interpretation for failing to replicate an original effect is that it did exist, but the replication attempt was unlikely to find it either because it was underpowered (Etz & Vandekerckhove, 2016), based on an improper definition of the replicandum (Wong & Steiner, 2018), or had a low a-priori replication chance due to other artifacts (e.g., Fiedler & Prager, 2018;D. J. Stanley & Spence, 2014).…”