Fast Subset Scan for Spatial Pattern Detection

Abstract

Summary. We propose a new 'fast subset scan' approach for accurate and computationally efficient event detection in massive data sets. We treat event detection as a search over subsets of data records, finding the subset which maximizes some score function. We prove that many commonly used functions (e.g. Kulldorff's spatial scan statistic and extensions) satisfy the 'linear time subset scanning' property, enabling exact and efficient optimization over subsets. In the spatial setting, we demonstrate that proximity-constrained subset scans substantially improve the timeliness and accuracy of event detection, detecting emerging outbreaks of disease 2 days faster than existing methods.

Non-parametric scan statistics for event detection and forecasting in heterogeneous social media graphs

“…where U (S, αmax) refers to the union of {αmax} and the set of distinct p-values less than αmax in S. Because the BJ scan statistic satisfies the linear time subset scanning (LTSS) property [16], the subproblem…”

Community sense and response systems

“…The main idea is to evaluate all subsets of the data for possible events. Of course, enumerating all possible O (2 n ) subsets is infeasible for even moderately sized problems, so later variations impose constraints on space and time to reduce the number of subsets, which is often done by scanning only subsets of distinct sizes and shapes [85,86]. This approach reduces the complexity to O n 2 (and to O (n) in [86]) but may also impair the detection performance if important signals are not well captured by tested subsets.…”

“…Linear Time Subset Scanning was recently developed to further reduce the complexity [86]. It states that for any scoring functions G, e.g., the ratio of count to baseline G(…”

“…For comparison with scan statistics, we implement the subset scan with spatial "k-fixed neighbors" constraints as described in [86]. Each subset i consists of node V i and its k nearest neighbors.…”

“…We also apply the subset scan baseline SS-k [86] for reference. The parameter k is "optimally" selected based on the prior knowledge that on average between 10 and 30 nodes are activated in each event in the cascade model.…”