Probabilistic forecasting of replication studies

Abstract

Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in two of the four data sets. In the other two data sets, predictive performance was still substantially improved compared to the naive model which does not consider inflation and heterogeneity of effects. The results suggest that many of the estimates from the original studies were inflated, possibly caused by publication bias or questionable research practices, and also that some degree of heterogeneity between original and replication effects should be expected. Moreover, the results indicate that the use of statistical significance as the only criterion for replication success may be questionable, since from a predictive viewpoint, non-significant replication results are often compatible with significant results from the original study. The developed statistical methods as well as the data sets are available in the R package ReplicationSuccess.

Replication Power and Regression to The Mean

“…The adjustment barely shrinks large effect estimates from very convincing original studies but applies more shrinkage to less convincing ones. 6 For our simulated data, the adjustment reduces the mean squared prediction error from 1.87 to 1.56 (17%).…”

Discussion on the Meeting on ‘Signs and Sizes:Understanding and Replicating Statistical Findings’

“…Ioannidis ()). The investigation of consistency between original and replication study, as mentioned by Dr Mathur and Professor VanderWeele, is more in the spirit of estimation and perhaps best described through probabilistic forecasting of the replication result based on the original result (Bayarri and Mayoral, ; Patil et al ., ; Pawel and Held, ). In rare cases it may then happen that replication success is declared although the replication effect size is inconsistently large compared with the original effect size: a scenario mentioned by Professor Bird.…”

“…Regarding the latter Professor Bird mentions the popular rule of thumb to halve the original effect size in sample size calculations, as used in the recent social science replication project (Camerer et al ., ). A more principled approach is to estimate the shrinkage prior variance with the empirical Bayes method (Pawel and Held, ), which has connections to shrinkage methods in regression for optimal prediction (Copas, ).…”

Statistical Assessment of Replicability via Bayesian Model Criticism

“…The prior-predictive replication p-values are u-values [38,39] defined by the following proposition. The two-sided prior-predictive replication p-values are connected to the predictive interval checking procedures in replication assessment [14,33]. Specifically, the interval-checking procedures construct a (1−α)% predictive interval based on an assumed reproducible model and the original experiment.…”

“…Some are motivated by investigating effects of irreproducibility due to specific factors, e.g., publication bias/winner's curse [24,25,26,27,28,15] and batch effects [29,30,31], in specific settings. They utilize different inference strategies, e.g., model comparison [32,19], predictive interval checking [14,33], and deals with distinct application scenarios. This diverse body of work lays a solid foundation for us to summarize important statistical principles in replication assessment.…”