Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits

Abstract

Monte Carlo (MC) permutation test is considered the gold standard for statistical hypothesis testing, especially when standard parametric assumptions are not clear or likely to fail. However, in modern data science settings where a large number of hypothesis tests need to be performed simultaneously, it is rarely used due to its prohibitive computational cost. In genome-wide association studies, for example, the number of hypothesis tests m is around 10 6 while the number of MC samples n for each test could be greater than 10 8 , totaling more than nm=10 14 samples. In this paper, we propose Adaptive MC multiple Testing (AMT) to estimate MC p-values and control false discovery rate in multiple testing. The algorithm outputs the same result as the standard full MC approach with high probability while requiring only Õ( √ nm) samples. This sample complexity is shown to be optimal. On a Parkinson GWAS dataset, the algorithm reduces the running time from 2 months for full MC to an hour. The AMT algorithm is derived based on the theory of multiarmed bandits.

Semi-supervised multiple testing

“…Let us already note that plugging empirically-based p-values into the BH procedure is not new and has been widely explored in the literature, especially in a Monte Carlo framework, see, e.g., Guo and Peddada (2008), Sandve et al (2011), Gandy and Hahn (2014), Zhang et al (2019). However, while the same null sample is used to compute all p-values in our setting, most of the existing works focus on the case where m null samples are available, that is, each test uses a different sample, often generated via randomization process (e.g., permutations).…”

“…Another approach is used in Sandve et al (2011) by allocating the Monte Carlo budget (total number of Monte Carlo samples) according to the significance of the test statistics, itself extending an idea of Besag and Clifford (1991) for single testing. More recently, Zhang et al (2019) proposed to reduce the computation burden by following a bandit approach. While all these works are based on null training samples, the crucial difference is that our setting only relies on one null sample for all tests.…”