A New Standard for the Analysis and Design of Replication Studies

Abstract

Summary A new standard is proposed for the evidential assessment of replication studies. The approach combines a specific reverse Bayes technique with prior‐predictive tail probabilities to define replication success. The method gives rise to a quantitative measure for replication success, called the sceptical p‐value. The sceptical p‐value integrates traditional significance of both the original and the replication study with a comparison of the respective effect sizes. It incorporates the uncertainty of both the original and the replication effect estimates and reduces to the ordinary p‐value of the replication study if the uncertainty of the original effect estimate is ignored. The framework proposed can also be used to determine the power or the required replication sample size to achieve replication success. Numerical calculations highlight the difficulty of achieving replication success if the evidence from the original study is only suggestive. An application to data from the Open Science Collaboration project on the replicability of psychological science illustrates the methodology proposed.

The Harmonic Meanχ2-Test to Substantiate Scientific Findings

“…A second post‐marketing clinical trial is then often required to confirm or revoke the initial decision (Zhang et al ., ). This setting has much in common with replication studies that try to confirm original results in independent investigations (Held, ; Roes, ).…”

“…The two‐trials rule has conditional power as described by Goodman (), but with a discontinuity at 0.025. The power curves of the two harmonic mean χ 2 ‐tests (calculated with the results given in Held (), section ) are smooth, quickly approaching zero at p 1 =0.065 and p 1 =0.083. Both the combined and the pooled method have longer tails with non‐zero conditional power even for a larger p ‐value of the first study.…”

“…Here I propose a new approach to assess the overall evidence for a treatment effect based on the harmonic mean of the squared Z ‐scores:This form is motivated from the special case of n =2 successive studies, one original and one replication, where a reverse Bayes approach for the assessment of replication success has recently been described (Held, ). If the two studies have equal precision (i.e.…”

“…If the two studies have equal precision (i.e. sample size), the assessment of replication success does not depend on the order of the two studies and is based on the test statistic ; compare Held (), equation (9). Equation extends this to n studies with an additional multiplicative factor n 2 , which ensures that the null distribution of equation does not depend on n .…”

“…Contrary to the two‐trials rule, it leads to approval of one trial with p =0.026 and the other with p =0.001, but not to approval if both trials have p =0.024. The work is motivated from a recent proposal how to evaluate the success of replication studies (Held, ) and is based on the harmonic mean of the squared Z ‐scores. It can include weights for the individual studies and can be calibrated to ensure exact type I error control and to compute an overall p ‐value; see Section .…”