Sequential Event Detection Using Multimodal Data in Nonstationary Environments

Abstract

The problem of sequential detection of anomalies in multimodal data is considered. The objective is to observe physical sensor data from CCTV cameras, and social media data from Twitter and Instagram to detect anomalous behaviors or events. Data from each modality is transformed to discrete time count data by using an artificial neural network to obtain counts of objects in CCTV images and by counting the number of tweets or Instagram posts in a geographical area. The anomaly detection problem is then formulated as a problem of quickest detection of changes in count statistics. The quickest detection problem is then solved using the framework of partially observable Markov decision processes (POMDP), and structural results on the optimal policy are obtained. The resulting optimal policy is then applied to real multimodal data collected from New York City around a 5K race to detect the race. The count data both before and after the change is found to be nonstationary in nature. The proposed mathematical approach to this problem provides a framework for event detection in such nonstationary environments and across multiple data modalities.

A Sequential Detection Theory for Statistically Periodic Random Processes

“…For the on-path camera, the average message count increases due to the event. This periodic pattern was also observed in other data that we collected in [32] and [33]. Thus, the problem of anomaly detection or change detection in traffic data can be posed as arXiv:1904.04239v2 [eess.SP] 13 Aug 2019 the problem of detecting deviations away from statistically periodic behavior.…”

“…where λ f > 0 is a penalty on the cost of false alarms. The above optimization problem can be stated as a problem in partially observable MDPs (POMDPs); see Section II-B, and also [1], [22], and [11]. Specifically, define p 0 = 0 and p n = P π (ν ≤ n|Y 1 , · · · , Y n ), for n ≥ 1.…”

“…If λ k = λ f and d k = 1, ∀k, then the problem in (18) reduces to the problem in (6). If the period T = 1 and the processes are i.i.d., then the problem in (18) can also be solved using classical MDP theory [1], [11], [17], [22]. However, for T > 1, the observation process is i.p.i.d.…”

“…Fig. 1: Cyber-physical system application: The average Instagram post counts for data collected in NYC in [32]. The first two figures show that the average counts have similar statistical properties across different days.…”

“…CCTV cameras in New York City (NYC). These counts were extracted from multi-modal traffic data (CCTV images, Twitter and Instagram messages) that we collected in NYC [32], [33]. We collected data during a 5K run that occurred on September 24, 2017.…”