Violence Recognition Based on Auditory-Visual Fusion of Autoencoder Mapping

Abstract

In the process of violence recognition, accuracy is reduced due to problems related to time axis misalignment and the semantic deviation of multimedia visual auditory information. Therefore, this paper proposes a method for auditory-visual information fusion based on autoencoder mapping. First, a feature extraction model based on the CNN-LSTM framework is established, and multimedia segments are used as whole input to solve the problem of time axis misalignment of visual and auditory information. Then, a shared semantic subspace is constructed based on an autoencoder mapping model and is optimized by semantic correspondence, which solves the problem of audiovisual semantic deviation and realizes the fusion of visual and auditory information on segment level features. Finally, the whole network is used to identify violence. The experimental results show that the method can make good use of the complementarity between modes. Compared with single-mode information, the multimodal method can achieve better results.

Lightweight mobile network for real-time violence recognition

“…Ref [ 24 ] introduces a semi-supervised approach into pre-trained I3D, which can improve accuracy by removing redundant data and focusing on useful visual information. Ref [ 38 ] uses CNN-LSTM to extract visual and auditory information simultaneously, then, a shared semantic subspace is constructed based on an autoencoder mapping model, which can fuse segment level features.…”