End-to-End Multimodal Emotion Recognition Using Deep Neural Networks

Tzirakis, Panagiotis; Trigeorgis, George; Nicolaou, Mihalis A.; Schuller, Björn; Zafeiriou, Stefanos

doi:10.1109/jstsp.2017.2764438

Cited by 552 publications

(315 citation statements)

References 42 publications

(48 reference statements)

Supporting

Mentioning

310

Contrasting

Unclassified

Order By: Relevance

“…Perez-Gaspar et al [53] extended the evolutionary computation of artificial neural networks and hidden Markov models in classifying four emotions (angry, sad, happy, and neutral) by combining the speech with facial expressions. Tzirakis et al [54] also fused the auditory and visual modalities using an end-to-end valencearousal emotion recognition model. They applied the CNN and ResNet models to extract features from speech and visual signals, respectively, which were then concatenated and fed into the LSTM model to accomplish the end-toend training manner.…”

Section: Multimodal Frameworkmentioning

confidence: 99%

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Zheng

2019

2019 9th International IEEE/EMBS Conference on Neural Engineering (NER)

View full text Add to dashboard Cite

Compared with the rich studies on the motor brain-computer interface (BCI), the recently emerging affective BCI presents distinct challenges since the brain functional connectivity networks involving emotion are not well investigated. Previous studies on emotion recognition based on electroencephalography (EEG) signals mainly rely on single-channel-based feature extraction methods. In this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public emotion EEG datasets: SEED, SEED-V, and DEAP. The strength feature achieves the best classification performance and outperforms the state-of-the-art differential entropy feature based on single-channel analysis for the EEG signals. The experimental results reveal that distinct functional connectivity patterns are exhibited for the five emotions of disgust, fear, sadness, happiness, and neutrality. Furthermore, we construct a multimodal emotion recognition model by combining the functional connectivity features from EEG and the features from eye movements or physiological signals using deep canonical correlation analysis. The classification accuracies of multimodal emotion recognition are 95.08 ± 6.42% on the SEED dataset, 84.51 ± 5.11% on the SEED-V dataset, and 85.34 ± 2.90% and 86.61 ± 3.76% for arousal and valence on the DEAP dataset, respectively. The results demonstrate the complementary representation properties of the EEG functional connectivity network features with eye movement data. In addition, we find that the brain networks constructed with fewer channels, i.e., 18 channels, achieve comparable performance with that of the 62-channel network with respect to multimodal emotion recognition and enable easier setups for BCI systems in real scenarios.Index Terms-Affective brain-computer interface, EEG, eye movement, brain functional connectivity network, multimodal emotion recognition. ! •

show abstract

Section: Multimodal Frameworkmentioning

confidence: 99%

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Zheng

2019

2019 9th International IEEE/EMBS Conference on Neural Engineering (NER)

View full text Add to dashboard Cite

show abstract

“…We compare the performance of proposed approach with raw waveform-based END2YOU network [23] for three sub challenges. In terms of U AR, proposed raw-CNN approach has a comparable performance with baseline for Crying task, whereas it outperforms the baseline for other two tasks, see Table 4.…”

Section: Results On Raw-waveform Cnnmentioning

confidence: 99%

“…We also compare the proposed approach with the corresponding end-to-end baseline. As the confidence scores are not available for the baseline system [23], it is not possible to use it for fusion techniques.…”

Section: Introductionmentioning

confidence: 99%

Implementing Fusion Techniques for the Classification of Paralinguistic Information

Vlasenko¹,

Sebastian²,

Kumar³

et al. 2018

Interspeech 2018

View full text Add to dashboard Cite

This work tests several classification techniques and acoustic features and further combines them using late fusion to classify paralinguistic information for the ComParE 2018 challenge. We use Multiple Linear Regression (MLR) with Ordinary Least Squares (OLS) analysis to select the most informative features for Self-Assessed Affect (SSA) sub-Challenge. We also propose to use raw-waveform convolutional neural networks (CNN) in the context of three paralinguistic sub-challenges. By using combined evaluation split for estimating codebook, we obtain better representation for Bag-of-Audio-Words approach. We preprocess the speech to vocalized segments to improve classification performance. For fusion of our leading classification techniques, we use weighted late fusion approach applied for confidence scores. We use two mismatched evaluation phases by exchanging the training and development sets, and this estimates the optimal fusion weight. Weighted late fusion provides better performance on development sets in comparison with baseline techniques. Raw-waveform techniques perform comparable to the baseline.

show abstract

“…A different approach is presented in [12] which considers using a CNN to extract features from the speech, whilst, in order to represent the visual information, a deep residual network ResNet-50 is used. The features from the beforementioned networks are concatenated and inserted into a twolayers Long Short-Term Memory (LSTM) module.…”

Section: B Emotion Recognition Using Visual and Audio Informationmentioning

confidence: 99%

Emotion Recognition System from Speech and Visual Information based on Convolutional Neural Networks

Ristea

Dutu²,

Rădoi

2019

2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD)

View full text Add to dashboard Cite

Emotion recognition has become an important field of research in the human-computer interactions domain. The latest advancements in the field show that combining visual with audio information lead to better results if compared to the case of using a single source of information separately. From a visual point of view, a human emotion can be recognized by analyzing the facial expression of the person. More precisely, the human emotion can be described through a combination of several Facial Action Units. In this paper, we propose a system that is able to recognize emotions with a high accuracy rate and in real time, based on deep Convolutional Neural Networks. In order to increase the accuracy of the recognition system, we analyze also the speech data and fuse the information coming from both sources, i.e., visual and audio. Experimental results show the effectiveness of the proposed scheme for emotion recognition and the importance of combining visual with audio data.

show abstract

End-to-End Multimodal Emotion Recognition Using Deep Neural Networks

Cited by 552 publications

References 42 publications

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Implementing Fusion Techniques for the Classification of Paralinguistic Information

Emotion Recognition System from Speech and Visual Information based on Convolutional Neural Networks

Contact Info

Product

Resources

About