The assessment of replication success based on relative effect size

Abstract

Replication studies are increasingly conducted in order to confirm original findings. However, there is no established standard how to assess replication success, and, in practice, many different approaches are used. The purpose of this paper is to refine and extend a recently proposed reverse-Bayes approach for the analysis of replication studies. We show how this method is directly related to the relative effect size, the ratio of the replication to the original effect estimate. This perspective leads to a new proposal to recalibrate the assessment of replication success, the golden level. The recalibration ensures that, for borderline significant original studies, replication success can only be achieved if the replication effect estimate is larger than the original one. Conditional power for replication success can then take any desired value if the original study is significant and the replication sample size is large enough. Compared to the standard approach to require statistical significance of both the original and replication study, replication success at the golden level offers uniform gains in project power and controls the type-I error rate if the replication sample size is not smaller than the original one. An application to data from four large replication projects shows that the new approach leads to more appropriate inferences, as it penalizes shrinkage of the replication estimate, compared to the original one, while ensuring that both effect estimates are sufficiently convincing on their own.

The Sceptical Bayes Factor for the Assessment of Replication Success

“…The sceptical p-value also penalises the case when the replication effect estimate shrinks as compared to the original one since it monotonically increases with decreasing relative effect estimate d (Held, 2020, section 3.1). Held et al (2022b) showed that replication success based on p S ≤ 𝛼 S is achieved when…”

“…For example, when z o = 2, 𝛼 = 0.025, and c = 2, the replication effect estimate needs to be d = 4.87 times larger than the original one. Therefore, Held et al (2022b) used Equation ( 14 The bottom-right plot in Figure 4 shows the success region for the recalibrated sceptical p-value. We see that increasing the precision of the replication study lowers the required minimum relative effect estimate d min as for all other methods.…”

“…One solution is to shrink the original effect estimate for power calculations (Held et al, 2022b;Pawel & Held, 2020), but we will not focus on this aspect here as this would not provide much more insight but simply lower the power curves of all methods.…”

“…with 𝜙(⋅) the standard normal density function. In a similar fashion one can compute the type I error rate of the sceptical p-value (see Section 3 in Held et al, 2022b), as well as the replication Bayes factor (Appendix G). The type I error rate of the two-trials rule is simply T1E = 2{1 − Φ(z 𝛾 )} 2 .…”

“…Despite most researchers agreeing on the importance of replication, there is currently no agreement on a statistical criterion for replication success. Instead, a variety of statistical methods, frequentist (Hedges & Schauer, 2019; Mathur & VanderWeele, 2020; Patil et al, 2016; Simonsohn, 2015), Bayesian (van Aert & van Assen, 2017; Bayarri & Mayoral, 2002a,b; Etz & Vandekerckhove, 2016; Harms, 2019; Johnson et al, 2017; Ly et al, 2018; Verhagen & Wagenmakers, 2014), and combinations thereof (Held, 2020; Held et al, 2022b; Pawel & Held, 2020) have been proposed to quantify replication success.…”