A Spatiotemporal Saliency Model for Video Surveillance

Abstract

A video sequence is more than a sequence of still images. It contains a strong spatial-temporal correlation between the regions of consecutive frames. The most important characteristic of videos is the perceived motion foreground objects across the frames. The motion of foreground objects dramatically changes the importance of the objects in a scene and leads to a different saliency map of the frame representing the scene. This makes the saliency analysis of videos much more complicated than that of still images. In this paper, we investigate saliency in video sequences and propose a novel spatiotemporal saliency model devoted for video surveillance applications. Compared to classical saliency models based on still images, such as Itti's model, and space-time saliency models, the proposed model is more correlated to visual saliency perception of surveillance videos. Both bottom-up and topdown attention mechanisms are involved in this model. Stationary saliency and motion saliency are, respectively, analyzed. First, a new method for background subtraction and foreground extraction is developed based on content analysis of the scene in the domain of video surveillance. Then, a stationary saliency model is setup based on multiple features computed from the foreground. Every feature is analyzed with a multi-scale Gaussian pyramid, and all the features conspicuity maps are combined using different weights. The stationary model integrates faces as a supplement feature to other low level features such as color, intensity and orientation. Second, a motion saliency map is calculated using the statistics of the motion vectors field. Third, both motion saliency map and stationary saliency map are merged based on center-surround framework defined by an approximated Gaussian function. The video saliency maps computed from our model have been compared to the gaze maps obtained from subjective experiments with SMI eye tracker for surveillance video sequences. The results show strong correlation between the output of the proposed spatiotemporal saliency model and the experimental gaze maps.

Saliency Aggregation: Does Unity Make Strength?

“…However, when the similarity score is high, we provide evidence that the loss is limited, compared to the upper bound for which the weights are estimated by minimizing the prediction error. For critical applications for which the relevance and robustness of the saliency map are fundamental such as video surveillance [23], object detection [13], clinical diagnostic [24], implementation of traffic sign [25], the conclusion of this study is interesting; the robustness of the prediction can be indeed enhanced by either averaging the saliency maps of the top 2 models or by considering a dedicated training dataset.…”

Spatio-temporal Saliency Detection in Dynamic Scenes Using Local Binary Patterns

“…The two maps are then fused to generate space-time saliency map. In a similar way, Tong et al [4] proposed a saliency model which is used for video surveillance. The spatial map is computed based on low level features and the dynamic map is computed based on motion intensity, motion orientation and phase.…”

“…It is an important and fundamental research problem in neuroscience and psychology to investigate the mechanism of human visual system in selecting regions of interest in complex scenes. It has been an active research topic in computer vision research due to various applications such as object detection [1], image segmentation [2], robotic navigation and localization [3], video surveillance [4], object tracking [5], image re-targeting [6] and image/video compression [7].…”

“…Most of the existing spatio-temporal saliency models [4], [14], [13] use optical flow methods to process the motion information. In these methods, motion intensity of each pixel is computed and the final saliency map represents the pixels which are moving against the background.…”

Auditory Saliency Classification Using EEG Data

“…However, when interacting with reality or observing a video, humans are not only driven by visually salient stimuli but also by auditory salient ones. While many computational models are available for the case of visual saliency maps [6,7,8], fewer models exist for acoustic attention [9]. Within this context, the aim of this work is to investigate if it is possible to classify salient audio events starting from EEG data.…”