Instantaneous threat detection based on a semantic representation of activities, zones and trajectories

Abstract

Threat detection is a challenging problem, because threats appear in many variations and differences to normal behaviour can be very subtle. In this paper, we consider threats on a parking lot, where theft of a truck's cargo occurs. The theft takes place in very different forms, in the midst of many people who pose no threat. The threats range from explicit, e.g., a person attacking the truck driver, to implicit, e.g., somebody loitering and then fiddling with the exterior of the truck in order to open it. Our goal is a system that is able to recognize a threat instantaneously as they develop. Typical observables of the threats are a person's activity, presence in a particular zone, and the trajectory. The novelty of this paper is an encoding of these threat observables in a semantic, intermediate-level representation, based on low-level visual features that have no intrinsic semantic meaning themselves. The semantic representation encodes the notions of trajectories, zones and activities. The aim of this representation is to bridge the semantic gap between the low-level tracks and motion and the higher-level notion of threats. In our experiments, we demonstrate that our semantic representation is more descriptive for threat detection than directly using low-level features. We find that a person's activities are the most important elements of this semantic representation, followed by the person's trajectory. The proposed threat detection system is very accurate: 96.6% of the tracks are correctly interpreted, when considering the temporal context.

Automatic detection of suspicious behavior of pickpockets with track-based features in a shopping mall

“…Future work may include the improvement of threat recognition by combining track-based analysis with other features (e.g. action recognition for local motion [9][10] or re-identification information for long-term analysis [5] [7]) or by extending it to other forms of complex threatening behavior [2][26] [27]. Recently, we started a project in collaboration with the Royal Marechaussee and Qubit Visual Intelligence at Amsterdam's Schiphol Airport, where we intend to detect activities such as people falling on the ground and theft of bags amid large crowds.…”

Video content analysis on body-worn cameras for retrospective investigation

“…For example, sterile-zone monitoring alerts operators for activities in industrial zones that should be abandoned at night. More advanced analysis is emerging to find suspects [7] and to detect suspicious behavior [9]. Besides fixed CCTV cameras we see an increasing number of sensors on mobile 'platforms', such as UAVs, vehicles, police officers and civilians.…”

Focus-of-attention for human activity recognition from UAVs

“…wave), shapes (e.g., dig), and movement (e.g., run). Therefore, we use motion features and adopt the approach as taken by [22] to associate motion features to tracks, and performing a track-based prediction as in [5,24] of the human activity that is being displayed.…”

“…In systems #1, #2 and #3 all features for a given image sequence are fed at once to the human activity recognition algorithm. Systems #4 and #5 interpret each track individually, see e.g., [5,24], based on its features. System #4 does so for each track, whereas system #5 only interprets tracks that are designated as human.…”