Interactive Martingale Tests for the Global Null

Abstract

Global null testing is a classical problem going back about a century to Fisher's and Stouffer's combination tests. In this work, we present simple martingale analogs of these classical tests, which are applicable in two distinct settings: (a) the online setting in which there is a possibly infinite sequence of p-values, and (b) the batch setting, where one uses prior knowledge to preorder the hypotheses. Through theory and simulations, we demonstrate that our martingale variants have higher power than their classical counterparts even when the preordering is only weakly informative. Finally, using a recent idea of "masking" p-values, we develop a novel interactive test for the global null that can take advantage of covariates and repeated user guidance to create a data-adaptive ordering that achieves higher detection power against structured alternatives.

Familywise Error Rate Control by Interactive Unmasking

“…It then progressively unmasks missing bits to help the interaction. A series of interactive tests are developed following this idea: Lei and Fithian [2018] and Lei et al [2017] proposed the masking idea and an interactive test with FDR control; Duan et al [2019] developed an interactive test with global Type 1 error control (an error metric less stringent than FDR and FWER); this work presents an interactive test with FWER control. At a high level, masking and interactive testing achieve a confirmatory conclusion while providing an exploratory framework.…”