Bounding the resampling risk for sequential Monte Carlo implementation of hypothesis tests

Kim, Hyune Ju

doi:10.1016/j.jspi.2010.01.003

Cited by 11 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Expected runtimes also hold true for the confidence sequences of Robbins (1970); Lai (1976) employed in the methods of Gandy and Hahn (2016); Ding et al (2018), as they do for the confidence sequences of Darling and Robbins (1967a,b) and the binomial confidence intervals of Armitage (1958) employed in Fay et al (2007); Gandy (2009). The results of this article do not apply to the push-out design of Fay and Follmann (2002) and the B-value design of Kim (2010), which both achieve a bounded resampling risk without confidence statements on the p-values.…”

Section: Introductionmentioning

confidence: 79%

On the expected runtime of multiple testing algorithms with bounded error

Hahn¹

2018

Preprint

View full text Add to dashboard Cite

Consider the testing of multiple hypotheses in the setting where the p-values of all hypotheses are unknown and thus have to be approximated using Monte Carlo simulations. One class of algorithms published in the literature for this scenario ensures guarantees on the correctness of their testing result (for instance, a bound on the resampling risk) through the computation of confidence statements on all approximated p-values. This article focuses on the expected runtime of such algorithms. It is known that in the aforementioned Monte Carlo scenario, computing a decision for a single hypothesis tested at a fixed threshold requires an infinite expected runtime. The article extends this result to multiple hypotheses through the following three main contributions. In applications relying on the decisions of multiple hypotheses computed with a Bonferroni-type threshold, all but two hypotheses can be decided in finite expected runtime. This result does not extend to applications which require full knowledge of all individual decisions (for instance, step-up or step-down procedures), in which case no algorithm can guarantee even a single decision in finite expected runtime. Nevertheless simulations show that in practice, the number of pending decisions typically remains low.

show abstract

Section: Introductionmentioning

confidence: 79%

On the expected runtime of multiple testing algorithms with bounded error

Hahn¹

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…The process can therefore be interrupted after some criterion has been reached. Various such criteria have been proposed ( Andrews and Buchinsky, 2000 , Davidson and MacKinnon, 2000 , Fay and Follmann, 2002 , Fay et al, 2007 , Gandy, 2009 , Kim, 2010 , Sandve et al, 2011 , Gandy and Rubin-Delanchy, 2013 , Ruxton and Neuhäuser, 2013 ), and of particular interest is the interruption after a predefined number n of exceedances T j ⁎ ⩾ T has been found. Weaker effects will quickly be exceeded after a few random shufflings, whereas stronger effects require insistence in doing more shufflings until exceedances are found.…”

Section: Theorymentioning

confidence: 99%

Faster permutation inference in brain imaging

et al. 2016

View full text Add to dashboard Cite

Permutation tests are increasingly being used as a reliable method for inference in neuroimaging analysis. However, they are computationally intensive. For small, non-imaging datasets, recomputing a model thousands of times is seldom a problem, but for large, complex models this can be prohibitively slow, even with the availability of inexpensive computing power. Here we exploit properties of statistics used with the general linear model (GLM) and their distributions to obtain accelerations irrespective of generic software or hardware improvements. We compare the following approaches: (i) performing a small number of permutations; (ii) estimating the p-value as a parameter of a negative binomial distribution; (iii) fitting a generalised Pareto distribution to the tail of the permutation distribution; (iv) computing p-values based on the expected moments of the permutation distribution, approximated from a gamma distribution; (v) direct fitting of a gamma distribution to the empirical permutation distribution; and (vi) permuting a reduced number of voxels, with completion of the remainder using low rank matrix theory. Using synthetic data we assessed the different methods in terms of their error rates, power, agreement with a reference result, and the risk of taking a different decision regarding the rejection of the null hypotheses (known as the resampling risk). We also conducted a re-analysis of a voxel-based morphometry study as a real-data example. All methods yielded exact error rates. Likewise, power was similar across methods. Resampling risk was higher for methods (i), (iii) and (v). For comparable resampling risks, the method in which no permutations are done (iv) was the absolute fastest. All methods produced visually similar maps for the real data, with stronger effects being detected in the family-wise error rate corrected maps by (iii) and (v), and generally similar to the results seen in the reference set. Overall, for uncorrected p-values, method (iv) was found the best as long as symmetric errors can be assumed. In all other settings, including for familywise error corrected p-values, we recommend the tail approximation (iii). The methods considered are freely available in the tool PALM — Permutation Analysis of Linear Models.

show abstract

“…where the dependence of g i on p i , α and c is omitted for notational simplicity. The probability in (1) is also called the resampling risk, a popular error measure which many algorithms published in the literature on Monte Carlo hypothesis testing aim to control (Davidson and MacKinnon, 2000;Fay and Follmann, 2002;Fay et al, 2007;Gandy, 2009;Kim, 2010;Ding et al, 2018). The total expected number of misclassifications which occur when allocating k = (k 1 , .…”

Section: Formulation Of the Problemmentioning

confidence: 99%

Optimal allocation of Monte Carlo simulations to multiple hypothesis tests

Hahn

2019

Stat Comput

View full text Add to dashboard Cite

Multiple hypothesis tests are often carried out in practice using p-value estimates obtained with bootstrap or permutation tests since the analytical p-values underlying all hypotheses are usually unknown. This article considers the allocation of a pre-specified total number of Monte Carlo simulations K ∈ N (i.e., permutations or draws from a bootstrap distribution) to a given number of m ∈ N hypotheses in order to approximate their p-values p ∈ [0, 1] m in an optimal way, in the sense that the allocation minimises the total expected number of misclassified hypotheses. A misclassification occurs if a decision on a single hypothesis, obtained with an approximated p-value, differs from the one obtained if its p-value was known analytically. The contribution of this article is threefold: Under the assumption that p is known and K ∈ R, and using a normal approximation of the Binomial distribution, the optimal real-valued allocation of K simulations to m hypotheses is derived when correcting for multiplicity with the Bonferroni correction, both when computing the p-value estimates with or without a pseudo-count. Computational subtleties arising in the former case will be discussed. Second, with the help of an algorithm based on simulated annealing, empirical evidence is given that the optimal integer allocation is likely of the same form as the optimal real-valued allocation, and that both seem to coincide asympotically. Third, an empirical study on simulated and real data demonstrates that a recently proposed sampling algorithm based on Thompson sampling asympotically mimics the optimal (real-valued) allocation when the p-values are unknown and thus estimated at runtime.

show abstract

Bounding the resampling risk for sequential Monte Carlo implementation of hypothesis tests

Cited by 11 publications

References 7 publications

On the expected runtime of multiple testing algorithms with bounded error

On the expected runtime of multiple testing algorithms with bounded error

Faster permutation inference in brain imaging

Optimal allocation of Monte Carlo simulations to multiple hypothesis tests

Contact Info

Product

Resources

About