Discussion on the Meeting on ‘Signs and Sizes:Understanding and Replicating Statistical Findings’

Hutton, Jane; Diggle, Peter J.; Bird, Sheila M.; Hennig, Christian; Longford, Nick; Mathur, Maya B.; VanderWeele, Tyler J.; Ioannidis, John P. A.; Chai, Christine P.; Dowe, David L.; Ferguson, John; Fitz-Simon, Nicola; Friede, Tim; Röver, Christian; Grieve, Andrew P.; Kumar, Kuldeep; Ly, Alexander; Mansmann, Ulrich; Mateu, Jorge; Matthews, Robert; Neuenschwander, Beat; Zwahlen, Marcel; Pericchi, Luis Raúl; Roes, Kit C. B.; Senn, Stephen; Wagenmakers, Eric‐Jan; Rice, Kenneth; Krakauer, Chloe; Bonnett, Tyler; Held, Leonhard

doi:10.1111/rssa.12544

Cited by 5 publications

(1 citation statement)

References 85 publications

(89 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For this reason, we extended the reverse-Bayes procedure from Held (2020b) to use Bayes factors for the purpose of quantifying evidence. This extension was suggested by Consonni (2019) and Pericchi (2020) independently in response to Held (2020b).…”

Section: Introductionmentioning

confidence: 60%

The Sceptical Bayes Factor for the Assessment of Replication Success

Pawel

Held

2022

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

Replication studies are increasingly conducted but there is no established statistical criterion for replication success. We propose a novel approach combining reverse‐Bayes analysis with Bayesian hypothesis testing: a sceptical prior is determined for the effect size such that the original finding is no longer convincing in terms of a Bayes factor. This prior is then contrasted to an advocacy prior (the reference posterior of the effect size based on the original study), and replication success is declared if the replication data favour the advocacy over the sceptical prior at a higher level than the original data favoured the sceptical prior over the null hypothesis. The sceptical Bayes factor is the highest level where replication success can be declared. A comparison to existing methods reveals that the sceptical Bayes factor combines several notions of replicability: it ensures that both studies show sufficient evidence against the null and penalises incompatibility of their effect estimates. Analysis of asymptotic properties and error rates, as well as case studies from the Social Sciences Replication Project show the advantages of the method for the assessment of replicability.

show abstract

Section: Introductionmentioning

confidence: 60%

The Sceptical Bayes Factor for the Assessment of Replication Success

Pawel

Held

2022

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

show abstract

Christine P. Chai’s contribution to the Discussion of ‘Safe testing’ by Grünwald, De Heide, and Koolen

Chai

2024

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

The assessment of replication success based on relative effect size

2022

View full text Add to dashboard Cite

Replication studies are increasingly conducted in order to confirm original findings. However, there is no established standard how to assess replication success, and, in practice, many different approaches are used. The purpose of this paper is to refine and extend a recently proposed reverse-Bayes approach for the analysis of replication studies. We show how this method is directly related to the relative effect size, the ratio of the replication to the original effect estimate. This perspective leads to a new proposal to recalibrate the assessment of replication success, the golden level. The recalibration ensures that, for borderline significant original studies, replication success can only be achieved if the replication effect estimate is larger than the original one. Conditional power for replication success can then take any desired value if the original study is significant and the replication sample size is large enough. Compared to the standard approach to require statistical significance of both the original and replication study, replication success at the golden level offers uniform gains in project power and controls the type-I error rate if the replication sample size is not smaller than the original one. An application to data from four large replication projects shows that the new approach leads to more appropriate inferences, as it penalizes shrinkage of the replication estimate, compared to the original one, while ensuring that both effect estimates are sufficiently convincing on their own.

show abstract

Discussion on the Meeting on ‘Signs and Sizes:Understanding and Replicating Statistical Findings’

Cited by 5 publications

References 85 publications

The Sceptical Bayes Factor for the Assessment of Replication Success

The Sceptical Bayes Factor for the Assessment of Replication Success

Christine P. Chai’s contribution to the Discussion of ‘Safe testing’ by Grünwald, De Heide, and Koolen

The assessment of replication success based on relative effect size

Contact Info

Product

Resources

About