Interactive martingale tests for the global null

Duan, Boyan; Ramdas, Aaditya; Balakrishnan, Sivaraman; Wasserman, Larry

doi:10.1214/20-ejs1790

Cited by 8 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…, P n , for example, by choosing two subsets with similar distribution of the P i in order for the isotonic regression estimator to extrapolate well. Systematic sampling approaches to increase the power have already been applied by [11] for testing treatment effects.…”

Section: Discussionmentioning

confidence: 99%

A Safe Hosmer-Lemeshow Test

Henzi,

Puke,

Dimitriadis

et al. 2023

The New England Journal of Statistics in Data Science

View full text Add to dashboard Cite

This article proposes an alternative to the Hosmer-Lemeshow (HL) test for evaluating the calibration of probability forecasts for binary events. The approach is based on e-values, a new tool for hypothesis testing. An e-value is a random variable with expected value less or equal to one under a null hypothesis. Large e-values give evidence against the null hypothesis, and the multiplicative inverse of an e-value is a p-value. Our test uses online isotonic regression to estimate the calibration curve as a ‘betting strategy’ against the null hypothesis. We show that the test has power against essentially all alternatives, which makes it theoretically superior to the HL test and at the same time resolves the well-known instability problem of the latter. A simulation study shows that a feasible version of the proposed eHL test can detect slight miscalibrations in practically relevant sample sizes, but trades its universal validity and power guarantees against a reduced empirical power compared to the HL test in a classical simulation setup. We illustrate our test on recalibrated predictions for credit card defaults during the Taiwan credit card crisis, where the classical HL test delivers equivocal results.

show abstract

Section: Discussionmentioning

confidence: 99%

A Safe Hosmer-Lemeshow Test

Henzi,

Puke,

Dimitriadis

et al. 2023

The New England Journal of Statistics in Data Science

View full text Add to dashboard Cite

show abstract

“…Cousins (2007) provides further examples and discussion of tests based on combining p-values while Owen (2009) contains a specialized review. In fact, methods based on suitably combining p-values and related quantities have received continued interest in the era of high-dimensional statistics as witnessed by, e.g., the recent works of Duan et al (2020) and Vovk and Wang (2020a,b).…”

Section: Examples Of Tests With Acceptance Region Satisfyingmentioning

confidence: 99%

“…Important early work includes Fisher (1934), Tippett (1931), Pearson (1933), Stouffer et al (1949) and Simes (1986). For a review of the classic literature see Cousins (2007), more recent contributions are Owen (2009), Duan et al (2020) and Vovk and Wang (2020a,b). It is crucial to highlight here that many of the above mentioned tests are consistent against strictly more alternatives than the LR test, i.e., they dominate the LR test in terms of their consistency properties.…”

Section: Introductionmentioning

confidence: 99%

Superconsistency of Tests in High Dimensions

Kock¹,

Preinerstorfer²

2021

Preprint

View full text Add to dashboard Cite

To assess whether there is some signal in a big database, aggregate tests for the global null hypothesis of no effect are routinely applied in practice before more specialized analysis is carried out. Although a plethora of aggregate tests is available, each test has its strengths but also its blind spots. In a Gaussian sequence model, we study whether it is possible to obtain a test with substantially better consistency properties than the likelihood ratio (i.e., Euclidean norm based) test. We establish an impossibility result, showing that in the high-dimensional framework we consider, the set of alternatives for which a test may improve upon the likelihood ratio test -that is, its superconsistency points -is always asymptotically negligible in a relative volume sense. * MSC 2020 subject classifications: 62F05 (Primary); 62F03, 62C20 (Secondary).

show abstract

“…Important early works include Tippett (1931), Pearson (1933), Fisher (1934), Stouffer et al (1949), andSimes (1986). For a review of the classic literature, see Cousins (2007), more recent contributions are Owen (2009), Duan et al (2020), and Wang (2020, 2021). It is crucial to highlight here that many of the above mentioned tests are consistent against strictly more alternatives than the LR test, i.e., they dominate the LR test in terms of their consistency properties; indeed, this is the main motivation of the power enhancement principle.…”

Section: Introductionmentioning

confidence: 99%

Superconsistency of Tests in High Dimensions

Kock

Preinerstorfer²

2022

Econom. Theory

View full text Add to dashboard Cite

To assess whether there is some signal in a big database, aggregate tests for the global null hypothesis of no effect are routinely applied in practice before more specialized analysis is carried out. Although a plethora of aggregate tests is available, each test has its strengths but also its blind spots. In a Gaussian sequence model, we study whether it is possible to obtain a test with substantially better consistency properties than the likelihood ratio (LR; i.e., Euclidean norm-based) test. We establish an impossibility result, showing that in the high-dimensional framework we consider, the set of alternatives for which a test may improve upon the LR test (i.e., its superconsistency points) is always asymptotically negligible in a relative volume sense.

show abstract

Interactive martingale tests for the global null

Cited by 8 publications

References 23 publications

A Safe Hosmer-Lemeshow Test

A Safe Hosmer-Lemeshow Test

Superconsistency of Tests in High Dimensions

Superconsistency of Tests in High Dimensions

Contact Info

Product

Resources

About