The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition

Abstract

Recently, the study of emotion recognition has received increasing attentions by the rapid development of noninvasive sensor technologies, machine learning algorithms and compute capability of computers. Compared with single modal emotion recognition, the multimodal paradigm introduces complementary information for emotion recognition. Hence, in this work, we presented a decision level fusion framework for detecting emotions continuously by fusing the Electroencephalography (EEG) and facial expressions. Three types of movie clips (positive, negative, and neutral) were utilized to elicit specific emotions of subjects, the EEG and facial expression signals were recorded simultaneously. The power spectrum density (PSD) features of EEG were extracted by time-frequency analysis, and then EEG features were selected for regression. For the facial expression, the facial geometric features were calculated by facial landmark localization. Long short-term memory networks (LSTM) were utilized to accomplish the decision level fusion and captured temporal dynamics of emotions. The results have shown that the proposed method achieved outstanding performance for continuous emotion recognition, and it yields 0.625±0.029 of concordance correlation coefficient (CCC). From the results, the fusion of two modalities outperformed EEG and facial expression separately. Furthermore, different numbers of time-steps of LSTM was applied to analyze the temporal dynamic capturing.INDEX TERMS Continuous emotion recognition, EEG, facial expressions, signal processing, decision level fusion, temporal dynamics.

Development and Progress in Sensors and Technologies for Human Emotion Recognition

“…In Figure 4 , we depict the process of EEG based human emotion recognition discussed in [ 92 ]. Several approaches have used EEG for emotion recognition [ 93 , 94 ]. For instance, the proposal [ 95 ] uses EEG to collect peripheral signals.…”

Hierarchical fusion of visual and physiological signals for emotion recognition

“…The representation ability of the dimensional model is more reliable and accurate than in the discrete model. Some researchers treat emotion recognition as continuous problem (Soleymani et al 2015;Sen and Sert 2018;Li et al 2019). To simplify the problem, other researchers usually treat the dimension prediction as a classification task, such as two-class (low/high), threeclass (lower/middle/higher), and four-class (quadrants of arousal/valence space) (Gunes and Schuller 2013).…”

“…Yohanes et al (2012) proposed to extract the EEG feature using discrete wavelet transform coefficients to utilize the temporal information. Li et al (2019) employed power spectrum density to extract features. Peng et al (2019) also explored using phase lag index to construct function connectivity to diagnose depression.…”

Identification of Suitable Biomarkers for Stress and Emotion Detection for Future Personal Affective Wearable Sensors

“…Happy and Routray [8] used image processing to detect emotional states in facial expression. Li et al [9] merged facial image processing with electroencephalography (EEG) for improved emotional state detection, indicating that affective systems benefit from being multimodal. Yang et al [10] demonstrated emotion detection through speech for AI-based home assistants.…”