Dimensional emotion recognition based on two stream CNN fusion attention mechanism

Abstract

Aiming at the problem that discrete emotion recognition cannot depict continuous emotion changes, in order to capture high-level dimensional emotional information, this paper integrates attention mechanism into the two stream CNN model and proposes a Two Stream Convolutional Neural Network with Shared and Global attention mechanism (TSCNN-SGA) .TSCNN-SGA uses the same structure of CNN network structure to extract the static stream of expression images and dynamic stream of expression sequences features respectively, firstly, in the dynamic and static dual flow feature extraction network, the output feature map of the previous convolution layer group is used to cascade to calculate the shared attention weight of the next layer group, secondly, the two stream convolution feature map with shared attention is cascaded, the attention weights of different positions are mapped onto the cascaded feature map and weighted, finally, the shared weight matrix in the convolution end of TSCNN-SSA and the global attention mechanism after the two stream feature cascade work together to obtain the depth space-time feature, which is input to the bidirectional long-short time network to obtain the final dimensional sentiment prediction value. Compared with different baseline methods, the average value of the proposed method's concordance correlation coefficient (CCC) in the arousal-valence space reached 0.576, which can effectively identify dimensional emotions.