On methods for correcting for the look-elsewhere effect in searches for new physics

Abstract

The search for new significant peaks over a energy spectrum often involves a statistical multiple hypothesis testing problem. Separate tests of hypothesis are conducted at different locations over a fine grid producing an ensemble of local p-values, the smallest of which is reported as evidence for the new resonance. Unfortunately, controlling the false detection rate (type I error rate) of such procedures may lead to excessively stringent acceptance criteria. In the recent physics literature, two promising statistical tools have been proposed to overcome these limitations. In 2005, a method to "find needles in haystacks" was introduced by Pilla et al. [1], and a second method was later proposed by Gross and Vitells [2] in the context of the "look-elsewhere effect" and trial factors. We show that, although the two methods exhibit similar performance for large sample sizes, for relatively small sample sizes, the method of Pilla et al. leads to an artificial inflation of statistical power that stems from an increase in the false detection rate. This method, on the other hand, becomes particularly useful in multidimensional searches, where the Monte Carlo simulations required by Gross and Vitells are often unfeasible. We apply the methods to realistic simulations of the Fermi Large Area Telescope data, in particular the search for dark matter annihilation lines. Further, we discuss the counter-intuitive scenario where the look-elsewhere corrections are more conservative than much more computationally efficient corrections for multiple hypothesis testing. Finally, we provide general guidelines for navigating the tradeoffs between statistical and computational efficiency when selecting a statistical procedure for signal detection.

Combined collider constraints on neutralinos and charginos

“…Approximate procedures can be applied in lower dimensions, e.g. [145][146][147], however, they are are mainly aimed at reducing the number of pseudodata realisations that are required to perform the trial correction, which is not the issue here. Our problem is instead that obtaining sufficiently good sampling of the possible signal predictions in the EWMSSM is hard.…”

Magnetic Field Disorder and Faraday Effects on the Polarization of Extragalactic Radio Sources

“…However, we have performed a total 90 tests, counting both KS and Spearman. To avoid the multiple hypothesis testing problem, we adopted the Bonferroni correction as discussed in Algeri et al (2016) and chose a conservative significance level threshold of p-value  -10 4 . We therefore rejected the null hypothesis of the KS or the Spearman rank tests only if the corresponding p-value is less than or equal to 10 −4 .…”

Search for Axionlike Dark Matter with a Liquid-State Nuclear Spin Comagnetometer

“…The cumulative distribution function F i (P ) of the power P at the ith frequency can be fitted with F i (P ) = 1 − A i exp(−B i P ), where A i and B i are the fitting parameters. With these parameters, we can derive the false-alarm thresholds corresponding to different global p values [31,53,54]. See the Supplemental Materials for detailed procedures [37].…”