Multimodal Fused Emotion Recognition About Expression-EEG Interaction and Collaboration Using Deep Learning

Abstract

The proposed emotion recognition model is based on the hierarchical long-short term memory neural network (LSTM) for video-electroencephalogram (Video-EEG) signal interaction. The inputs are facial-video and EEG signals from the subjects when they are watching the emotion-stimulated video. The outputs are the corresponding emotion recognition results. Facial-video features and corresponding EEG features are extracted based on a fully connected neural network (FC) at each time point. These features are fused through hierarchical LSTM to predict the key emotional signal frames at the next time point until the emotion recognition result is calculated at the last time point. Specially, a self-attention mechanism is applied to show the correlation of the stacked LSTM at different hierarchies. In this process, the ''selective focus'' is used to analyze the human-emotional temporal sequences in each model, which improves the utilization of the key spatial EEG signals. Moreover, the process includes the temporal attention mechanism to predict the key signal frame at next time point, which utilizes the key emotion data in temporal domain. The experimental results prove that the classification rate (CR) and F1-score of the proposed emotion recognition model are significantly increased by at least 2% and 0.015, respectively, compared to other methods.

[Retracted] Detection Method of Limb Movement in Competitive Sports Training Based on Deep Learning

“…Depth picture sequences have shown to be very valuable in the development of quick 3D human skeletal joint estimation. Many depth sensors with high sample rates and inexpensive prices have lately been launched, owing to the fast development of depthsensing technology [8]. Table 1 gives some common depth sensor information.…”

A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances

“…1) Effects of different signals (textual, audio, visual, or physiological) on unimodal affect recognition [152,194,244,289,327,331]; 2) Effects of modality combinations and fusion strategies on multimodal affective analysis [37,371,375,383,388,400]; 3) Effects of ML-based techniques [127,184,249,305,326] or DL-based methods [146,203,273,283,316,335] on affective computing; 4) Effects of some potential factors (e.g., released databases and performance metrics) on affective computing; 5) Applications of affective computing in real-life scenarios.…”

“…Soleymani et al [399] designed a framework of video-EEG based emotion detection using an LSTM-RNN and continuous conditional random fields. Wu et al [400] proposed a hierarchical LSTM with a self-attention mechanism to fuse the facial features and EEG features to calculate the final emotion. Yin et al [397] proposed an efficient end-to-end framework of EDA-music fused emotion recognition, denominating it as a 1-D residual temporal and channel attention network (RTCAN-1D).…”

“…The visual modality (facial expression, voice, gesture, posture, etc.) may also be integrated with multimodal physiological signals for visual-physiological affective analysis [397,400].…”

Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network

“…Feature extraction and fusion are the key steps in multimodal emotion recognition [3]. Li et al [4] proposed a hierarchical modular neural network and applied it to multimodal emotion recognition.…”