Multiple Testing with the Structure-Adaptive Benjamini–Hochberg Algorithm

Abstract

Summary In multiple‐testing problems, where a large number of hypotheses are tested simultaneously, false discovery rate (FDR) control can be achieved with the well‐known Benjamini–Hochberg procedure, which a(0,1]dapts to the amount of signal in the data, under certain distributional assumptions. Many modifications of this procedure have been proposed to improve power in scenarios where the hypotheses are organized into groups or into a hierarchy, as well as other structured settings. Here we introduce the ‘structure‐adaptive Benjamini–Hochberg algorithm’ (SABHA) as a generalization of these adaptive testing methods. The SABHA method incorporates prior information about any predetermined type of structure in the pattern of locations of the signals and nulls within the list of hypotheses, to reweight the p‐values in a data‐adaptive way. This raises the power by making more discoveries in regions where signals appear to be more common. Our main theoretical result proves that the SABHA method controls the FDR at a level that is at most slightly higher than the target FDR level, as long as the adaptive weights are constrained sufficiently so as not to overfit too much to the data—interestingly, the excess FDR can be related to the Rademacher complexity or Gaussian width of the class from which we choose our data‐adaptive weights. We apply this general framework to various structured settings, including ordered, grouped and low total variation structures, and obtain the bounds on the FDR for each specific setting. We also examine the empirical performance of the SABHA method on functional magnetic resonance imaging activity data and on gene–drug response data, as well as on simulated data.

AdaPT: An Interactive Procedure for Multiple Testing with Side Information

“…We compare AdaPT against 12 other methods: (a)SeqStep with parameter C =2 (Barber and Candès, ); (b)ForwardStop (Grazier G’Sell et al ., ); (c)the accumulation test with the HingeExp function and parameter C =2 (Li and Barber, ); (d)Adaptive SeqStep with s = q and λ =1− q (Lei and Fithian, ); (e)the BH procedure (Benjamini and Hochberg, ); (f)Storey's BH procedure with threshold λ =0.5 (Storey et al ., ); (g)the Barber–Candès method (Barber and Candès, ; Arias‐Castro and Chen, ); (h)SABHA with τ =0.5, ε =0.1 and stepwise constant weights, monotone, taking values in { ε ,1} (see section 4.1 of Li and Barber ()); (i)SABHA with τ =0.5, ε =0.1 and monotone weights, taking values in [ ε ,1] (see section 4.1 of Li and Barber ()); (j)IHW with the number of bins and folds set as default (Ignatiadis et al ., ); (k)an oracle version of IHW with the number of bins determined by maximizing the number of rejections; (l)an oracle version of independent filtering (IF) with the cut‐off determined by maximizing the number of rejections (Bourgon et al ., ). …”

“…To illustrate the power of AdaPT, we apply it to the GEOquery gene–dosage data (Davis and Meltzer, ), which have been analysed repeatedly as a benchmark for ordered testing procedures (Li and Barber, , ; Lei and Fithian, ). We use algorithm 2 with a beta mixture model (15) for the E‐step (see the on‐line appendix A.1.1 for details) and a gamma GLM with canonical link function for the M‐step.…”

“…Let and denote respectively the permutations of i ∈ [ n ] given by the stronger and weaker orderings. Further details on these two orderings can be found in Li and Barber (). We write the p ‐values, thus reordered, as and .…”

“…In summary, these plots show a strong data adaptivity of AdaPT, which can also learn the local structure of data while controlling the FDR. Moreover, it provides a quantitative way, by estimated signal strength, to evaluate the quality of ordering, which is the major concern in ordered testing problems (Li and Barber, ; Lei and Fithian, ).…”

“…Perhaps the procedure that is most closely related to ours is the structure‐adaptive BH algorithm SABHA of Li and Barber (). SABHA first censors the p ‐values below at a fixed level τ ( τ =0.5 in their simulations), leading to censored p ‐values p i 1 { p i > τ }.…”