Hierarchical Attention‐Based Multimodal Fusion Network for Video Emotion Recognition

Abstract

The context, such as scenes and objects, plays an important role in video emotion recognition. The emotion recognition accuracy can be further improved when the context information is incorporated. Although previous research has considered the context information, the emotional clues contained in different images may be different, which is often ignored. To address the problem of emotion difference between different modes and different images, this paper proposes a hierarchical attention-based multimodal fusion network for video emotion recognition, which consists of a multimodal feature extraction module and a multimodal feature fusion module. The multimodal feature extraction module has three subnetworks used to extract features of facial, scene, and global images. Each subnetwork consists of two branches, where the first branch extracts the features of different modes, and the other branch generates the emotion score for each image. Features and emotion scores of all images in a modal are aggregated to generate the emotion feature of the modal. The other module takes multimodal features as input and generates the emotion score for each modal. Finally, features and emotion scores of multiple modes are aggregated, and the final emotion representation of the video will be produced. Experimental results show that our proposed method is effective on the emotion recognition dataset.

Sparse Spatial-Temporal Emotion Graph Convolutional Network for Video Emotion Recognition

“…To evaluate the performance of the proposed method, we conduct experiments on four publicly available video emotion recognition datasets, namely, the MHED dataset [35], the HEIV dataset [36], the VideoEmotion-8 dataset [37], and the Ekman-6 dataset [20].…”

“…To evaluate the performance of the proposed method, we conduct experiments on four publicly available video emotion recognition datasets, namely, the MHED dataset [ 35 ], the HEIV dataset [ 36 ], the VideoEmotion-8 dataset [ 37 ], and the Ekman-6 dataset [ 20 ]. MHED [ 35 ]: the MHED dataset is composed of 1066 videos, which uses a training set of 638 videos and a testing set of 428 videos.…”

“…It uses six emotion categories “anger,” “disgust,” “fear,” “joy,” “sadness,” and “surprise,” defined by the psychologists Ekman and Friesen [ 38 ] to label human emotions in the video. HEIV [ 36 ]: the HEIV dataset is composed of 1012 videos with 607 training videos and 405 for testing. Six emotion categories, “anger,” “disgust,” “fear,” “joy,” “sadness,” and “surprise,” defined by the psychologists Ekman and Friesen [ 38 ], as well as neutral emotion, are used to label human emotions in the video.…”

Region Dual Attention-Based Video Emotion Recognition

“…We conduct experiments on five publicly available video emotion recognition data sets, namely the MHED data set [ 28 ], the HEIV data set [ 29 ], the ekman-6 data set [ 30 ], the videoemotion-8 data set [ 31 ], and the SFEW data set [ 32 ]. The MHED data set is composed of 1,066 videos that are manually downloaded from the network, and it uses a training set of 638 videos and a testing set of 428 videos.…”

Unified multi‐stage fusion network for affective video content analysis

“…To this end, recent studies designed neural networks and optimised model parameters [4]. Although previous researches [5][6][7][8][9][10][11][12][13] have achieved promising progress, it is still challenging to analyse the emotions induced by videos.…”