Safe Testing

Grünwald, Peter; Heide, Rianne de; Koolen, Wouter M.

doi:10.1109/ita50056.2020.9244948

Cited by 64 publications

(84 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So any choice will be a balancing act. How to perform this balancing act is an important topic for further research (Grünwald et al., 2019).…”

Section: Betting Games As Statistical Modelsmentioning

confidence: 99%

Testing by Betting: A Strategy for Statistical and Scientific Communication

Shafer

2021

Journal of the Royal Statistical Society Series A: Statistics in Society

View full text Add to dashboard Cite

The most widely used concept of statistical inference-the p -value-is too complicated for effective communication to a wide audience (Gigerenzer, 2018 ; McShane & Gal, 2017 ). This paper introduces a simpler way of reporting statistical evidence: report the outcome of a bet against the null hypothesis. This leads to a new role for likelihood, to alternatives to power and confidence, and to a framework for meta-analysis that accommodates both planned and opportunistic testing of statistical hypotheses and probabilistic forecasts.Testing a hypothesized probability distribution by betting is straightforward. We select a nonnegative payoff and buy it for its hypothesized expected value. If this bet multiplies the money it risks by a large factor, we have evidence against the hypothesis, and the factor measures the strength of this evidence. Multiplying our money by 5 might merit attention; multiplying it by 100 or by 1000 might be considered conclusive.

show abstract

“…So any choice will be a balancing act. How to perform this balancing act is an important topic for further research (Grünwald et al., 2019).…”

Section: Betting Games As Statistical Modelsmentioning

confidence: 99%

Testing by Betting: A Strategy for Statistical and Scientific Communication

Shafer

2021

Journal of the Royal Statistical Society Series A: Statistics in Society

View full text Add to dashboard Cite

show abstract

“…Without sample splitting and with a fixed alternative distribution, the RIPR LRT has been recently studied (35). When PΘ is convex and the RIPR split LRT is implementable, meaning that it is computationally feasible to find the RIPR or evaluate its likelihood, then this test can be more powerful than the MLE split LRT.…”

Section: Misspecification and Convex Model Classesmentioning

confidence: 99%

“…Such confidence sequences have been developed under very general nonparametric, multivariate, matrix, and continuoustime settings using generalizations of the aforementioned supermartingale technique (39)(40)(41). The connections between anytime-valid P values, e values, safe tests, peeking, confidence sequences, and the properties of optional stopping and continuation have been explored recently (35,40,42,43). The connection to the present work is that when run sequentially, our universal (MLE or RIPR) split LRT yields an anytime-valid P value, an e value, and a safe test, which can be inverted to form universal confidence sequences and are valid under optional stopping and continuation, and these are simply because the underlying process of interest is bounded by a nonnegative (super)martingale.…”

Section: Anytime P Values and Confidence Sequencesmentioning

confidence: 99%

Universal inference

Wasserman

Ramdas

Balakrishnan

2020

Proc. Natl. Acad. Sci. U.S.A.

112

View full text Add to dashboard Cite

We propose a general method for constructing confidence sets and hypothesis tests that have finite-sample guarantees without regularity conditions. We refer to such procedures as “universal.” The method is very simple and is based on a modified version of the usual likelihood-ratio statistic that we call “the split likelihood-ratio test” (split LRT) statistic. The (limiting) null distribution of the classical likelihood-ratio statistic is often intractable when used to test composite null hypotheses in irregular statistical models. Our method is especially appealing for statistical inference in these complex setups. The method we suggest works for any parametric model and also for some nonparametric models, as long as computing a maximum-likelihood estimator (MLE) is feasible under the null. Canonical examples arise in mixture modeling and shape-constrained inference, for which constructing tests and confidence sets has been notoriously difficult. We also develop various extensions of our basic methods. We show that in settings when computing the MLE is hard, for the purpose of constructing valid tests and intervals, it is sufficient to upper bound the maximum likelihood. We investigate some conditions under which our methods yield valid inferences under model misspecification. Further, the split LRT can be used with profile likelihoods to deal with nuisance parameters, and it can also be run sequentially to yield anytime-valid P values and confidence sequences. Finally, when combined with the method of sieves, it can be used to perform model selection with nested model classes.

show abstract

“…Not to be confused with the e-values produced by safe tests(Grünwald, de Heide, & Koolen, 2020), where the e refers to expectation rather than evidence.…”

mentioning

confidence: 99%

A Critical Evaluation of the FBST ev for Bayesian Hypothesis Testing

Ly¹,

Wagenmakers²

2021

Preprint

View full text Add to dashboard Cite

he “Full Bayesian Significance Test e-value”, henceforth FBST ev, has received increasing attention across a range of disciplines including psychology. We show that the FBST ev leads to four problems: (1) the FBST ev cannot quantify evidence in favor of a null hypothesis and therefore also cannot discriminate “evidence of absence” from “absence of evidence”; (2) the FBST ev is susceptible to sampling to a foregone conclusion; (3) the FBST ev violates the principle of predictive irrelevance, such that it is affected by data that are equally likely to occur under the null hypothesis and the alternative hypothesis; (4) the FBST ev suffers from the Jeffreys-Lindley paradox in that it does not include a correction for selection. These problems also plague the frequentist p-value. We conclude that although the FBST ev may be an improvement over the p-value, it does not provide a reasonable measure of evidence against the null hypothesis.

show abstract

Safe Testing

Cited by 64 publications

References 36 publications

Testing by Betting: A Strategy for Statistical and Scientific Communication

Testing by Betting: A Strategy for Statistical and Scientific Communication

Universal inference

A Critical Evaluation of the FBST ev for Bayesian Hypothesis Testing

Contact Info

Product

Resources

About