Violence Detection in Videos Based on Fusing Visual and Audio Information

Abstract

Determining whether given video frames contain violent content is a basic problem in violence detection. Visual and audio information are useful for detecting violence included in a video, and are usually complementary; however, violence detection studies focusing on fusing visual and audio information are relatively rare. Therefore, we explored methods for fusing visual and audio information. We proposed a neural network containing three modules for fusing multimodal information: 1) attention module for utilizing weighted features to generate effective features based on the mutual guidance between visual and audio information; 2) fusion module for integrating features by fusing visual and audio information based on the bilinear pooling mechanism; and 3) mutual Learning module for enabling the model to learn visual information from another neural network with a different architecture. Experimental results indicated that the proposed neural network outperforms existing state-of-the-art methods on the XD-Violence dataset.

Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection

“…Finally, a mutual learning module is added to make the model learn visual information from another neural network with a different structure. Unlike Pang's method [26], we found that the violence detection accuracy of different kinds of videos is related to the change of optical flow. Therefore, our network added the optical flow feature to extract the motion features of objects and effectively solve the problems of short duration and weak action for the task of violence detection.…”

“…By using a sequence composed of multiple segments as an optimization unit, they reduced the probability of selection errors during training. Pang et al [ 26 ] further improved the algorithm on the basis of Wu et al [ 22 ], who focused on fusing audio and visual information. First, weighted features are used to generate effective features under the guidance of audio and visual information.…”

Bilinear pooling in video-QA: empirical challenges and motivational drift from neurological parallels

“…Problem with alignment of text and video: As we highlight in the second subsection of our related works, BLP has yielded great performance in video tasks where it fuses the visual features with non-textual features. Audio and visual feature fusion demonstrates impressive performance on action recognition( Hu et al, 2021 ), emotion recognition ( Zhou et al, 2021 ), and violence detection ( Pang et al, 2021 ). Likewise, different visual representations have thrived in RGBT tracking ( Xu et al, 2021 ), action recognition ( Deng et al, 2021 ) and video-QA on MSVD-QA ( Wang, Bao & Xu, 2021 ).…”

“… Hu et al (2021) use compact BLP to fuse audio and ‘visual long range’ features for human action recognition. Pang et al (2021) use MLB as part of an attention-based fusion for audio and visual features for violence detection in videos. Xu et al (2021) use BLP to fuse visual features from different channels in colour image (RGB) and thermal infrared tracking (TiR) i.e.…”

“…Furthermore, despite BLP’s history of success in text-image fusion in VQA, it has not yet gained such notoriety in video-QA. Though BLP methods have continued to perform well on video tasks when fusing vision and non-textual features ( Hu et al, 2021 ; Zhou et al, 2021 ; Pang et al, 2021 ; Xu et al, 2021 ; Deng et al, 2021 ; Wang, Bao & Xu, 2021 ; Deb et al, 2022 ; Sudhakaran, Escalera & Lanz, 2021 ), BLP has recently been overshadowed by other vision and textual feature fusion techniques in video-QA ( Kim et al, 2019 ; Li et al, 2019 ; Gao et al, 2019 ; Liu et al, 2021 ; Liang et al, 2019 ). In this paper, we aim to add a new perspective to the empirical and motivational drift in BLP.…”