Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Abstract

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

Replicability, Robustness, and Reproducibility in Psychological Science

“…Open Science Collaboration (2015) replicated 100 findings from 2008 issues of three psychology journals and observed 36% achieved significance in the same direction with effect sizes 49% as large as the original studies. "Multi-site replications" include the series titled "Many Labs" (Ebersole et al, , 2020Klein et al, 2014Klein et al, , 2018Klein et al, , 2019, registered replication reports primarily from the journal Advances in Methods and Practices in Psychological Science (Alogna et al, 2014;Bouwmeester et al, 2017;Cheung et al, 2016;Colling et al, 2020;Eerland et al, 2016;Hagger et al, 2016;McCarthy, Skowronski, et al, 2018;O'Donnell et al, 2018;Verschuere et al, 2018;Wagenmakers et al, 2016), papers from the Collaborative Replications and Education Project (Ghelfi et al, 2020;Leighton et al, 2018;Wagge et al, 2018), and other similar efforts (Dang et al, 2021;ManyBabies Consortium, 2020;Mccarthy et al, 2020;McCarthy, Hartnett, et al, 2018;Moran et al, 2020;Schweinsberg et al, 2016). Collectively (n = 77), 56% of multisite replications reported statistically significant evidence in the same direction with effect sizes 53% as large as the original studies ( Figure 1).…”

“…Speculative appeals to context sensitivity are common in response to failures to replicate (Cesario, 2014;Crisp et al, 2014;Dijksterhuis, 2018;Ferguson et al, 2014;Gilbert et al, 2016;Schnall, 2014;Schwarz & Strack, 2014;Shih & Pittinsky, 2014), but empirical demonstrations of a failure to replicate that was surmised to be due to failure to address context sensitivity then being "restored" in follow-up tests addressing the presumed sensitivity have yet to be observed. Further, there are multiple examples of presumed context sensitivity failing to occur or account for replication failures when examined directly (Ebersole, Atherton, et al, 2016;Ebersole et al, 2020;Klein et al, 2014Klein et al, , 2018. Heterogeneity is sometimes observed in replication studies, but it is usually modest and insufficient to make a replicable phenomenon appear or disappear based on factors that would not have been anticipated in advance of conducting the studies (Baribault et al, 2018;Klein et al, 2014Klein et al, , 2018Olsson-Collentine et al, 2020).…”

“…Predictions of replication outcomes across three methods: Surveys, Elicitations, and Prediction Markets. The figure aggregates 123 prediction-replication pairs for which there are both survey and market predictions and outcomes from five different prediction projects (Camerer et al, 2016(Camerer et al, , 2018Dreber et al, 2015;Ebersole et al, 2020;Forsell et al, 2019), and 25 elicitations for 25 replications (Wintle et al 2021). Probabilities of replication were computed on a 0 to 100 scale for all three methods, and all three sets of predictions were performed by experts.…”

“…In addition, Ebersole et al (2020) added prediction markets to the Many Labs 5 project, where researchers were asked to predict the replication outcomes of the 10 RP:P protocols and the 10 revised protocols. While two revised protocols replicated the original effect, the markets (consisting mainly of psychologists) believed that all but two other versions (RP:P and revised protocol) for another study would not replicate.…”

“…In the figures we show here we have combined the pooled results from Gordon et al (2021) and the results from Ebersole et al (2020). This leads to a sample of 123 predictionreplication pairs for which there are both survey and market predictions.…”