Beyond the two-trials rule: Type-I error control and sample size planning with the sceptical $p$-value

Abstract

We introduce a novel statistical framework to study replicability which simultaneously offers overall Type-I error control, an assessment of compatibility and a combined confidence region. The approach is based on a recently proposed reverse-Bayes method for the analysis of replication success. We show how the method can be recalibrated to obtain a family of combination tests for two studies with exact overall Type-I error control. The approach avoids the double dichotomization for significance of the two-trials rule and has larger project power to detect existing effects. It gives rise to a p-value function which can be used to compute a confidence region for the underlying true effect. If the effect estimates are compatible, the resulting confidence interval is similar to the meta-analytic one, but in the presence of conflict, the confidence region splits into two disjoint intervals. The proposed approach is applied to data from the Experimental Economics Replication Project.