Optimal detection of a jump in the intensity of a Poisson process or in a density with likelihood ratio statistics

Abstract

We consider the problem of detecting a ‘bump’ in the intensity of a Poisson process or in a density. We analyze two types of likelihood ratio‐based statistics, which allow for exact finite sample inference and asymptotically optimal detection: The maximum of the penalized square root of log likelihood ratios (‘penalized scan’) evaluated over a certain sparse set of intervals and a certain average of log likelihood ratios (‘condensed average likelihood ratio’). We show that penalizing the square root of the log likelihood ratio — rather than the log likelihood ratio itself — leads to a simple penalty term that yields optimal power. The thus derived penalty may prove useful for other problems that involve a Brownian bridge in the limit. The second key tool is an approximating set of intervals that is rich enough to allow for optimal detection, but which is also sparse enough to allow justifying the validity of the penalization scheme simply via the union bound. This results in a considerable simplification in the theoretical treatment compared with the usual approach for this type of penalization technique, which requires establishing an exponential inequality for the variation of the test statistic. Another advantage of using the sparse approximating set is that it allows fast computation in nearly linear time. We present a simulation study that illustrates the superior performance of the penalized scan and of the condensed average likelihood ratio compared with the standard scan statistic.

Multiscale change-point segmentation: beyond step functions

“…Examples of normal systems include the highly redundant system I 0 of all intervals whose end-points lie on the grid {i/n} n−1 i=0 (suggested by e.g. Siegmund and Yakir, 2000;Dümbgen and Spokoiny, 2001;Frick et al, 2014) of order O(n 2 ), and less redundant but still asymptotically efficient systems (Davies and Kovac, 2001;Walther, 2010;Rivera and Walther, 2013), typically of order O(n log n). Remarkably, there are even normal systems with cardinality of order O(n), such as the dyadic partition system…”

The essential histogram

“…We will show that on the event A n , (17) implies (18) √ n |H(I) − F n (I)| F n (I)(1 − F n (I)) ≥c n +c 2 n 2 nF n (I)(1 − F n (I)) 1(F n (I) < H(I)) ev., uniformly in H and I, wherec n := (F n (I)) + √ b n /4. Hence Lemma S1 (b,c) gives Rivera and Walther (2013).…”

“…To avoid lengthy technical work we will prove the theorem using J = { all intervals in R} in the definition of C n (α). The technical work in Rivera and Walther (2013) shows that the approximating set of intervals used in Section 2 is fine enough so that the optimality results continue to hold with that approximating set.…”

Automated and Robust Quantification of Colocalization in Dual-Color Fluorescence Microscopy: A Nonparametric Statistical Approach

“…This means that the number of distinct values among {τ (X (j ) , Y (k ) ) : (j , k ) ∈ π −1 i (j, k)} is not very large. We can then apply union bound, (18) and Lemma 1 to get…”