Time-uniform Chernoff bounds via nonnegative supermartingales

Howard, Steven R.; Ramdas, Aaditya; McAuliffe, Jon; Sekhon, Jasjeet S.

doi:10.1214/18-ps321

Cited by 48 publications

(56 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where step (i) follows by Ville's inequality (38,39), a timeuniform version of Markov's inequality for nonnegative supermartingales. Naturally, this test does not have to start at t = 1 when only one sample is available, meaning that we can set M0 = M1 = • • • = Mt 0 = 1 for the first t0 steps and then begin the updates.…”

Section: Anytime P Values and Confidence Sequencesmentioning

confidence: 99%

“…Such confidence sequences have been developed under very general nonparametric, multivariate, matrix, and continuoustime settings using generalizations of the aforementioned supermartingale technique (39)(40)(41). The connections between anytime-valid P values, e values, safe tests, peeking, confidence sequences, and the properties of optional stopping and continuation have been explored recently (35,40,42,43).…”

Section: Anytime P Values and Confidence Sequencesmentioning

confidence: 99%

See 1 more Smart Citation

Universal inference

Wasserman

Ramdas

Balakrishnan

2020

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

112

View full text Add to dashboard Cite

We propose a general method for constructing confidence sets and hypothesis tests that have finite-sample guarantees without regularity conditions. We refer to such procedures as “universal.” The method is very simple and is based on a modified version of the usual likelihood-ratio statistic that we call “the split likelihood-ratio test” (split LRT) statistic. The (limiting) null distribution of the classical likelihood-ratio statistic is often intractable when used to test composite null hypotheses in irregular statistical models. Our method is especially appealing for statistical inference in these complex setups. The method we suggest works for any parametric model and also for some nonparametric models, as long as computing a maximum-likelihood estimator (MLE) is feasible under the null. Canonical examples arise in mixture modeling and shape-constrained inference, for which constructing tests and confidence sets has been notoriously difficult. We also develop various extensions of our basic methods. We show that in settings when computing the MLE is hard, for the purpose of constructing valid tests and intervals, it is sufficient to upper bound the maximum likelihood. We investigate some conditions under which our methods yield valid inferences under model misspecification. Further, the split LRT can be used with profile likelihoods to deal with nuisance parameters, and it can also be run sequentially to yield anytime-valid P values and confidence sequences. Finally, when combined with the method of sieves, it can be used to perform model selection with nested model classes.

show abstract

Section: Anytime P Values and Confidence Sequencesmentioning

confidence: 99%

Section: Anytime P Values and Confidence Sequencesmentioning

confidence: 99%

Universal inference

Wasserman

Ramdas

Balakrishnan

2020

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

112

View full text Add to dashboard Cite

show abstract

“…Theorems 1 to 3 and Lemma 2 are our key tools for constructing confidence sequences. All build upon the general framework for uniform exponential concentration introduced in Howard et al [25], which means our techniques apply in diverse settings: scalar, matrix and Banachspace-valued observations, with possibly unbounded support; self-normalized bounds applicable to observations satisfying weak moment or symmetry conditions; and continuous-time FIG. 1.…”

mentioning

confidence: 99%

Time-uniform, nonparametric, nonasymptotic confidence sequences

et al. 2021

Self Cite

View full text Add to dashboard Cite

A confidence sequence is a sequence of confidence intervals that is uniformly valid over an unbounded time horizon. Our work develops confidence sequences whose widths go to zero, with nonasymptotic coverage guarantees under nonparametric conditions. We draw connections between the Cramér-Chernoff method for exponential concentration, the law of the iterated logarithm (LIL) and the sequential probability ratio test-our confidence sequences are time-uniform extensions of the first; provide tight, nonasymptotic characterizations of the second; and generalize the third to nonparametric settings, including sub-Gaussian and Bernstein conditions, self-normalized processes and matrix martingales. We illustrate the generality of our proof techniques by deriving an empirical-Bernstein bound growing at a LIL rate, as well as a novel upper LIL for the maximum eigenvalue of a sum of random matrices. Finally, we apply our methods to covariance matrix estimation and to estimation of sample average treatment effect under the Neyman-Rubin potential outcomes model.

show abstract

“…NMs are likelihood ratios, capital processes and admissible e‐values (Ramdas et al., 2020; Shafer et al., 2011). Beyond the singleton parametric setting above,

frakturP

‐NMs can be constructed for several rich non‐parametric classes

frakturP

(Howard et al., 2020). Not only is a likelihood ratio dQ / dP a pointwise P ‐NM, but it is known that every composite

frakturP

‐NM can be represented as a likelihood ratio dQ / dP for every

P \in P

(Ramdas et al., 2020; Shafer et al., 2011).…”

mentioning

confidence: 99%

“…(Here Q plays the role of Shafer’s implied alternative for a bet against P , and the

frakturP

‐NM is thus a capital process.) These

frakturP

‐NMs can also be used to construct non‐parametrically valid ‘confidence sequences’ (Howard et al., 2020), measure‐theoretic analogues of Shafer’s warranty sets. In fact, composite

frakturP

‐NMs (coupled with the optional stopping theorem) yield admissible ‘safe’ e‐values in a concrete sense (Grünwald et al., 2019; Ramdas et al., 2020).…”

mentioning

confidence: 99%

Aaditya Ramdas’s Contribution to the Discussion of ‘Testing by Betting: A Strategy for Statistical and Scientific Communication’ by Glenn Shafer

Ramdas

2021

Journal of the Royal Statistical Society Series A: Statistics in Society

Self Cite

View full text Add to dashboard Cite

The most widely used concept of statistical inference-the p -value-is too complicated for effective communication to a wide audience (Gigerenzer, 2018 ; McShane & Gal, 2017 ). This paper introduces a simpler way of reporting statistical evidence: report the outcome of a bet against the null hypothesis. This leads to a new role for likelihood, to alternatives to power and confidence, and to a framework for meta-analysis that accommodates both planned and opportunistic testing of statistical hypotheses and probabilistic forecasts.Testing a hypothesized probability distribution by betting is straightforward. We select a nonnegative payoff and buy it for its hypothesized expected value. If this bet multiplies the money it risks by a large factor, we have evidence against the hypothesis, and the factor measures the strength of this evidence. Multiplying our money by 5 might merit attention; multiplying it by 100 or by 1000 might be considered conclusive.

show abstract

Time-uniform Chernoff bounds via nonnegative supermartingales

Cited by 48 publications

References 61 publications

Universal inference

Universal inference

Time-uniform, nonparametric, nonasymptotic confidence sequences

Aaditya Ramdas’s Contribution to the Discussion of ‘Testing by Betting: A Strategy for Statistical and Scientific Communication’ by Glenn Shafer

Contact Info

Product

Resources

About