AdaPT: An Interactive Procedure for Multiple Testing with Side Information

Abstract

Summary We consider the problem of multiple‐hypothesis testing with generic side information: for each hypothesis Hi we observe both a p‐value pi and some predictor xi encoding contextual information about the hypothesis. For large‐scale problems, adaptively focusing power on the more promising hypotheses (those more likely to yield discoveries) can lead to much more powerful multiple‐testing procedures. We propose a general iterative framework for this problem, the adaptive p‐value thresholding procedure which we call AdaPT, which adaptively estimates a Bayes optimal p‐value rejection threshold and controls the false discovery rate in finite samples. At each iteration of the procedure, the analyst proposes a rejection threshold and observes partially censored p‐values, estimates the false discovery proportion below the threshold and proposes another threshold, until the estimated false discovery proportion is below α. Our procedure is adaptive in an unusually strong sense, permitting the analyst to use any statistical or machine learning method she chooses to estimate the optimal threshold, and to switch between different models at each iteration as information accrues. We demonstrate the favourable performance of AdaPT by comparing it with state of the art methods in five real applications and two simulation studies.

AdaFDR: a Fast, Powerful and Covariate-Adaptive Approach to Multiple Hypothesis Testing

“…In addition, the covariate patterns learned by AdaFDR are shown in Figure 3b for the Bottomly data and the Pasilla data, and in Supplementary Figure 2c for the airway data. The alternative hypotheses are more likely to occur when the expression levels are high, consistent with previous findings 12,13,24 .…”

“…The default parameters of AdaFDR are used for every experiment in this paper, both real data analysis and simulations, without any tuning. In addition to the experiments, we theoretically prove that AdaFDR controls FDP with high probability when the null p-values, conditional on the covariates, are independently distributed and stochastically greater than the uniform distribution, a standard assumption also made by related literature 13,23,24 . Related works.…”

“…The key assumption is that the null p-values remain uniform regardless of the covariate value while others, including the alternative p-values and the likelihood for the hypotheses to be true null/alternative, may have arbitrary dependencies on the covariate. This is a standard assumption in the literature 12,23,24 . For example, in the case of AAF, the null p-values are uniformly distributed independent of AAF since the gene expression has no association with the SNP under the null hypothesis.…”

“…Such estimator has been used in recent works 24,27,48,49 and yields a conservative estimate of the true number of false discoveries (FD), in the sense that its expected value is larger than that of the true FD under mild assumptions (Lemma 1 in Supplementary Materials). Furthermore, FDP can be simply estimated as FDP(t) = FD(t) D(t) .…”

Multiple Testing with the Structure-Adaptive Benjamini–Hochberg Algorithm

“…Barber and Candès's (2015) selective SeqStep modifies the goal by rejecting only those p-values in the first stretch of the list that also lie below some threshold, and Lei and Fithian's (2016) adaptive SeqStep method finds a substantial increase in power by adapting to the proportion of nulls. In other settings, the ordering of the tests might not be fixed in advance, but might be determined over time by interacting with information revealed in the data set; Lei and Fithian's (2018) AdaPT method and the STAR method of Lei et al (2017) propose algorithms testing in an order that is determined interactively without loss of FDR control.…”