Estimating means of bounded random variables by betting

Abstract

This paper derives confidence intervals (CI) and time-uniform confidence sequences (CS) for the classical problem of estimating an unknown mean from bounded observations. We present a general approach for deriving concentration bounds, that can be seen as a generalization and improvement of the celebrated Chernoff method. At its heart, it is based on a class of composite nonnegative martingales, with strong connections to testing by betting and the method of mixtures. We show how to extend these ideas to sampling without replacement, another heavily studied problem. In all cases, our bounds are adaptive to the unknown variance, and empirically vastly outperform existing approaches based on Hoeffding or empirical Bernstein inequalities and their recent supermartingale generalizations by Howard et al. [2021]. In short, we establish a new state-of-the-art for four fundamental problems: CSs and CIs for bounded means, when sampling with and without replacement.

The e-posterior

“…[51], this may be viewed as ‘overkill’ and poses practical (and unnecessary) difficulties for Bayesian non-parametric inferences especially when the samples are small. By contrast, Waudby-Smith & Ramdas [52] provide an e-collection that can be used to learn the mean of a sequence of bounded random variables based on a small sample without any assumptions on the underlying random variables except boundedness and existence of a common mean.…”

Assessing the calibration of multivariate probabilistic forecasts

“…If one aims to construct powerful tests for the global null hypothesis of exchangeability of the multivariate ensemble members and the observation, it appears sensible to use a limited number of pre‐rank functions that provide complementary information, and to combine the resulting e‐values into one e‐value by predictable mixing; we refer readers to Waudby‐Smith and Ramdas (2023) and Casgrain et al (2023), where predictable mixing of test martingales is discussed and is referred to as so‐called betting strategies. The same reasoning applies in the case of general multivariate predictive distributions; here, the null hypothesis would be that the forecasts are auto‐calibrated; see Appendix .…”

Anytime-Valid Confidence Sequences in an Enterprise A/B Testing Platform

“…This method has excellent properties in practice but relies on a parametric model for validity, requiring strong assumptions that are unrealistic in practice. Subsequent work focused on extending CSs to richer nonparametric problems, [11] such as those for bounded random variables [15,20,31]. See Ramdas et al [22] for a more detailed survey.…”

“…See Ramdas et al [22] for a more detailed survey. Most of the CS literature focuses on non-asymptotic methods which have three major disadvantages even for fixed-horizon settings: (a) they require strong assumptions, such as a parametric model or known moment generating functions [11,23,27,30,31], (b) they are typically wider than asymptotic methods based on the central limit theorem, and (c) they take different forms for different problems, whereas the central limit theorem yields a universal and closedform (trivial-to-compute) expression. The CSs of Waudby-Smith et al [29] overcomes these issues, at the cost of satisfying anytime validity only in an asymptotic sense [29], and for this reason, we adopt their AsympCS framework in the present work.…”

“…As we discuss in Section 2, there are multiple approaches that allow for optional stopping, ranging from group sequential methods [9,19,21] (commonly employed in clinical trials), to Bayesian methods [7,25], and more recently, a burgeoning literature on "anytimevalid" methods [11][12][13]31] which are based on martingale techniques. Anytime-valid procedures have several desirable properties -allowing for continuous monitoring, adaptive stopping, or continuation, while controlling type-I error at all times.…”