How large should the next study be? Predictive power and sample size requirements for replication studies

Abstract

We use information derived from over 40K trials in the Cochrane Collaboration database of systematic reviews (CDSR) to compute the replication probability, or predictive power of an experiment given its observed (two-sided) P-value. We find that an exact replication of a marginally significant result with P = .05 has less than 30% chance of again reaching significance. Moreover, the replication of a result with P = .005 still has only 50% chance of significance. We also compute the probability that the direction (sign) of the estimated effect is correct, which is closely related to the type S error of Gelman and Tuerlinckx. We find that if an estimated effect has P = .05, there is a 93% probability that its sign is correct. If P = .005, then that probability is 99%. Finally, we compute the required sample size for a replication study to achieve some specified power conditional on the p-value of the original study. We find that the replication of a result with P = .05 requires a sample size more than 16 times larger than the original study to achieve 80% power, while P = .005 requires at least 3.5 times larger sample size.These findings confirm that failure to replicate the statistical significance of a trial does not necessarily indicate that the original result was a fluke.

Proposal of a Selection Protocol for Replication of Studies in Sports and Exercise Science

“…Along with the high positive result rate in the field [33], this protocol focuses on statistically significant effects for reasons of feasibility. An attempt to replicate a non-significant effect (where p > 0.05) would require infeasible sample sizes more than 16 times the original to obtain 80% replication power [41]. There is a risk that replication failures might be overestimated by using original p-values due to the wide sampleto-sample variability and influence of sample size [47,48].…”

“…There is a possibility that doubling the original sample size will lead to underpowered replication studies. If an original study has a p-value of 0.03, the replication power is estimated to be 50% when using this method [41]. Although this could be highly informative for the statistical power of our field and potential shrinkage of effect size estimates [20].…”

Bayesian approaches to designing replication studies.

“…SSD for replication studies comes with unique opportunities and challenges; the data from the original study can be used to inform SSD, and at the same time the analysis of replication success based on the original and replication study is typically different from an analysis of a single study for which traditional SSD methodology was developed. Since the design of replication studies should be aligned with the planned analysis, a small literature has emerged that specifically deals with power calculations and SSD for replication studies (Anderson & Kelley, 2022; Anderson & Maxwell, 2017; Bayarri & Mayoral, 2002; Goodman, 1992; Hedges & Schauer, 2021; Held, 2020; Micheloud & Held, 2022; Pawel & Held, 2022; Senn, 2002; van Zwet & Goodman, 2022). However, most of these articles only deal with selected analysis methods and data models.…”

Ten simple rules for designing and conducting undergraduate replication projects

“…Entire textbooks and a wealth of tutorials have been written about choosing the best statistical tests for a given research problem and data type; however, some specific resources might come in handy both for the supervisor and for the students involved in a replication. For example, it is worth considering issues such as statistical power in the context of replication [30][31][32], as well as best practices in the design and analysis of replication studies [33,34]. General resources about statistics in science can be beneficial to students who need a refresher on specific aspects of research methods; see for example [35] for an introductory online textbook on statistics, [36] for a great resource on the design of experiments and observational studies, [37] for a thorough guide covering various aspects of data analysis, or [38] for a primer on biostatistics using R. The following webpage (https://bookdown.org/home/tags/statistics/) includes these resources and many more to choose from depending on the needs of the project and of your students-all free and accessible to all.…”