Efficient Spatio-Temporal Modeling Methods for Real-Time Violence Recognition

Abstract

Violence recognition is challenging since recognition must be performed on videos acquired by a lot of surveillance cameras at any time or place. It should make reliable detections in real time and inform surveillance personnel promptly when violent crimes take place. Therefore, we focus on efficient violence recognition for real-time and on-device operation, for easy expansion into a surveillance system with numerous cameras. In this paper, we propose a novel violence detection pipeline that can be combined with the conventional 2-dimensional Convolutional Neural Networks (2D CNNs). In particular, framegrouping is proposed to give the 2D CNNs the ability to learn spatio-temporal representations in videos. It is a simple processing method to average the channels of input frames and group three consecutive channelaveraged frames as an input of the 2D CNNs. Furthermore, we present spatial and temporal attention modules that are lightweight but consistently improve the performance of violence recognition. The spatial attention module named Motion Saliency Map (MSM) can capture salient regions of feature maps derived from the motion boundaries using the difference between consecutive frames. The temporal attention module called Temporal Squeeze-and-Excitation (T-SE) block can inherently highlight the time periods that are correlated with a target event. Our proposed pipeline brings significant performance improvements compared to the 2D CNNs followed by the Long Short-Term Memory (LSTM) and much less computational complexity than existing 3D-CNN-based methods. In particular, MobileNetV3 and EfficientNet-B0 with our proposed modules achieved state-of-the-art performance on six different violence datasets. Our codes are available at https://github.com/ahstarwab/Violence_Detection. INDEX TERMS Real-time violence detection, efficient spatio-temporal attention, efficient convolution method for spatio-temporal modeling

Efficient Two-Stream Network for Online Video Action Segmentation

“…It requires more parameters and FLOPS compared to the 2D convolution since the expanded kernels stride in time as well as space. Frame-grouping is proposed in our previous work [30] to give 2D CNNs the ability to learn spatio-temporal representations in videos. We make each three-channel frame into a single-channel frame z t ∈ R H×W by averaging across the channel axis and group three consecutive frames to learn spatio-temporal representations in a video with the conventional 2D-CNN backbones as follows:…”

“…We apply frame-grouping in the first convolutional layer of the networks to get a three-frame temporal receptive field. As discussed in [30], frame-grouping is effective to model shortterm dynamics. It can efficiently deal with some actions that can be captured in a short duration.…”

“…MobileNetV2 with a multiplier of 1.0 was used for the backbone network. Although many video action segmentation models run at 15 fps, enough time gaps between adjacent frames are required for the frame-grouping to secure a sufficient temporal receptive field with three consecutive frames as reported in our previous work [30]. Therefore, the 50Salads and GTEA datasets were downsampled to 7.5 fps, and the window size was set to 30 frames to have the number of input frames as a multiple of three.…”

“…For example, to distinguish between pushing and pulling something, the direction of hands or objects can be critical factors. Accordingly, we used frame-grouping [30] that is a method using three channel-averaged images as an input of 2D CNN for extracting motion information. Although the frame-grouping is designed to encode short-term dynamics like optical flow inputs [15], [31], [32], it does not require any computational overhead for feature extraction.…”

“…1) We introduce a temporal module that can encode motion information efficiently by extending the previous work [30] from violence recognition to video action segmentation. The previous work used frame-grouping module to model short-term dynamics by averaging channels and grouping three frames as an input of network.…”