Predicting the replicability of social science lab experiments

Abstract

We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.

Economic Inequality and Masculinity–Femininity: The Prevailing Perceived Traits in Higher Unequal Contexts Are Masculine

“…However, it is important to be cautious when interpreting Study 2’s three-way interaction because of (a) the relatively small sample size of Study 2 and (b) the fact that interactive effects are typically harder to replicate than main effects (Altmejd et al, 2019).…”

Probabilistic forecasting of replication studies

“…It is worth pointing out that in our forecasting approach the link between original and replication study is based only on information from the original study and the sample size of the replication study. This is fundamentally different from approaches where the link between original and replication study is estimated from a training sample of past original and replication study pairs, as for example done recently by Altmejd et al [14]. Since in our approach no replication estimates are used to estimate any parameter, all evaluations presented in this paper provide "out-of-sample" performance measures, thereby eliminating the need to split the data and perform cross-validation.…”

“…Making forecasts about an uncertain future is a common human desire and central for decision making in science and society [10,11]. There have been many attempts to forecast the outcomes of replication studies based on the results from the original studies [5,7,[12][13][14][15]. This is interesting for various reasons: First, a forecast of how likely a replication will be "successful" according to some criterion (e. g. an effect estimate reaches statistical significance) can help to assess the credibility of the original finding in the first place and inform the decision whether a replication study should be conducted at all.…”

Probabilistic forecasting of replication studies

“…It is worth pointing out that in our forecasting approach the link between original and replication study is based only on information from the original study and the sample size of the replication study. This is fundamentally different from approaches where the link between original and replication study is estimated from a training sample of past original and replication study pairs, as for example done recently by Altmejd et al (2019). Since in our approach no replication estimates are used to estimate any parameter, all evaluations presented in this paper provide "out-of-sample" performance measures, thereby eliminating the need to split the data and perform cross-validation.…”

“…Making forecasts about an uncertain future is a common human desire and central for decision making in science and society (Gneiting, 2008;Gneiting and Katzfuss, 2014). There have been many attempts to forecast the outcomes of replication studies based on the results from the original studies (Dreber et al, 2015;Patil et al, 2016;Camerer et al, 2016Camerer et al, , 2018Altmejd et al, 2019;Forsell et al, 2019). This is interesting for various reasons: First, a forecast of how likely a replication will be "successful" according to some criterion (e. g. an effect estimate reaches statistical significance) can help to assess the credibility of the original finding in the first place and inform the decision whether a replication study should be conducted at all.…”