A Unified Framework for Robust and Efficient Hotspot Detection in Smart Cities

Abstract

Given N geo-located point instances (e.g., crime or disease cases) in a spatial domain, we aim to detect subregions (i.e., hotspots) that have a higher probability density of generating such instances than the others. Hotspot detection has been widely used in a variety of important urban applications, including public safety, public health, urban planning, and equity, among others. The problem is challenging because its societal applications often have low tolerance for false positives and require significance testing that is computationally intensive. In related work, the spatial scan statistic introduced a likelihood ratio-based framework for hotspot evaluation and significance testing. However, it fails to consider the effect of spatial non-determinism, causing many missing detections. Our previous work introduced a non-deterministic normalization-based scan statistic to mitigate this issue. However, its robustness against false positives is not stably controlled. To address these limitations, we propose a unified framework that can improve the completeness of results without incurring more false positives. We also propose a reduction algorithm to improve the computational efficiency. Experiment results confirm that the unified framework can greatly improve the recall of hotspot detection without increasing the number of false positives, and the reduction algorithm can greatly reduce execution time.

Statistically-Robust Clustering Techniques for Mapping Spatial Hotspots: A Survey

“…Many of these standard formulations, including Kulldorff's likelihood ratio, have also been shown to be scale-sensitive. Specifically, these test statistics can be strongly biased towards small candidates [25,153,180,186,188] (albeit much…”

“…Another view of the average likelihood ratio approach is that it uses marginal likelihoods instead of maximum likelihoods [121], which may better utilize secondary cluster information as in the Bayesian test statistics [114,130,132]. In addition to the pure-scale interpretation of the bias, a spatial interpretation has been developed [186,188]. The analysis shows that the likelihoods or probabilities used to score a candidate in many approaches are based on a space bi-partition where one partition represents the candidate 𝐶 and the other for the outside.…”

“…A Revisit. There are two major reasons for selecting and testing only the optimal region in a single round [87,95,121,128,188,199]. First and most importantly, returning only the optimal region allows robust control over the false positive rate given a significance level 𝛼; otherwise, detecting multiple regions (e.g., candidates at different locations or with different sizes) at the same time may be susceptible to multi-testing, i.e., increasing the chance of having a type-I error.…”

“…As suggested by many studies, this maximization can be viewed as an automatic adjustment of significance levels for different groups of candidates (e.g., candidates of the same size) [152,153,180]. The intuition can also be easily explained from a data perspective [183,188]. Given 𝑀 datasets generated under a null hypothesis 𝐻 0 , having a critical value -a score threshold derived to determine a candidate's significance at the 𝛼 level -on the maximum score makes sure that a spurious cluster can only be falsely reported as a hotspot in smaller than 𝛼𝑀 datasets.…”

“…The possibility of using only a small set of samples from the entire dataset to construct upper bounds of p-values has been explored. Specifically, a reduction algorithm was developed in [188]. It shows that the original significance-enforced clustering problem can be mapped to a series of reduced problems (e.g., from 𝑁 to ⌈𝜌𝑁 ⌉ points) where the p-values of reduced clusters in these problems can be used to upper-bound the p-values of clusters in the original problem.…”