Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection

Abstract

Emotions are time varying affective phenomena that are elicited as a result of stimuli. Videos and movies in particular are made to elicit emotions in their audiences. Detecting the viewers' emotions instantaneously can be used to find the emotional traces of videos. In this paper, we present our approach in instantaneously detecting the emotions of video viewers' emotions from electroencephalogram (EEG) signals and facial expressions. A set of emotion inducing videos were shown to participants while their facial expressions and physiological responses were recorded. The expressed valence (negative to positive emotions) in the videos of participants' faces were annotated by five annotators. The stimuli videos were also continuously annotated on valence and arousal dimensions. Long-short-term-memory recurrent neural networks (LSTM-RNN) and continuous conditional random fields (CCRF) were utilized in detecting emotions automatically and continuously. We found the results from facial expressions to be superior to the results from EEG signals. We analyzed the effect of the contamination of facial muscle activities on EEG signals and found that most of the emotionally valuable content in EEG features are as a result of this contamination. However, our statistical analysis showed that EEG signals still carry complementary information in presence of facial expressions

Emotion Understanding Using Multimodal Information Based on Autobiographical Memories for Alzheimer’s Patients

“…The proposed quaternion based features improves the overall results by more than 1%. The proposed facial features also provide better F1 scores than the ones used in [20] in most of the classification scenarios. On the other hand, the results of the combined features are not always consistent in terms of which combination is the best one.…”

“…A leave one out approach and a k-fold cross validation is applied for all the participants in our database. These results are compared with the ones obtained using as features the suggested in [20]. Tables 2 and 3 show the F1 scores for all the modalities and both classifiers, SVM and gentleboost, respectively.…”

“…Considering that these sources are easy to fake [25] amongst other problems, EEG provides an extra source that solves problems such as falseness, illumination or speech impaired subjects. On the other hand, EEG signal deals with other challenges such as noise and biological and non-biological artifacts [20,30], such as electrooculogram (EOG), electromyogram (EMG) and electrocardiogram (ECG). Nevertheless, these biological artifacts are also affected by emotions and are expected to provide extra information to EEG signal for emotion recognition [20].…”

“…The work in [20] considers both types of data using the MAHNOB-HCI database [21]. The EEG signal was captured using 32 sensors and the power spectral density was extracted from overlapping one second windows.…”

“…For example, the study of speech and other acoustic cues in [18], body movements in [19],Electroencephalogram (EEG) in [25], facial expressions or combinations of previous ones such as speech and facial expressions in [22] or EEG and facial expressions in [20]. Our approach will focus on EEG and facial emotion detection.…”