Multimodal Emotion Recognition Using Cross-Modal Attention and 1D Convolutional Neural Networks

Emotion recognition is a topic of significant interest in assistive robotics due to the need to equip robots with the ability to comprehend human behavior, facilitating their effective interaction in our society. Consequently, efficient and dependable emotion recognition systems supporting optimal human-machine communication are required. Multi-modality (including speech, audio, text, images, and videos) is typically exploited in emotion recognition tasks. Much relevant research is based on merging multiple data modalities and training deep learning models utilizing low-level data representations. However, most existing emotion databases are not large (or complex) enough to allow machine learning approaches to learn detailed representations. This paper explores modalityspecific pre-trained transformer frameworks for self-supervised learning of speech and text representations for data-efficient emotion recognition while achieving state-of-the-art performance in recognizing emotions. This model applies feature-level fusion using nonverbal cue data points from motion capture to provide multimodal speech emotion recognition. The model was trained using the publicly available IEMOCAP dataset, achieving an overall accuracy of 77.58% for four emotions, outperforming state-of-the-art approaches

show abstract

“…Validation Accuracy Speech-only LSTM + Attention [24] GRU [25] Bi-LSTM [26] GRU + Attention [13] Bi-LSTM + Attention […”

Section: Modelmentioning

confidence: 99%

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Patamia

Acheampong

et al. 2021

2021 IEEE 2nd International Conference on Pattern Recognition and Machine Learning (PRML)

View full text Add to dashboard Cite

show abstract

“…Following prior works [13], we utilized a residual connection on audio and textual representation to keep the original structure of the data (H r a , H r t ). Then it is passed through a linear and a normalization layer.…”

Section: Cross-modal Robertamentioning

confidence: 99%

“…Previous methods have achieved good performance [8,9], there still exist key challenges in multimodal emotion recognition: different modalities depend on independent preprocessing and feature excavation designs due to heterogeneous space; and to make an applicable and generalizable model for both the individual modalities and fusion model, it is necessary to learn intra-and cross-modal interactions to reveal discriminative emotion content; and emotion is a subjective concept [10∼12]. Moreover, textual information and the associated context conveys more influential information in inferring the speaker's emotions [13], and plays a crucial role in inferring emotions in conversations. Unfortunately, the existing approaches cannot efficiently capture the textual emotional-relevant content in the fusion processing, and often the learning of intra-and crossmodal information will lose some semantic content [12,14].…”

Section: Introductionmentioning

confidence: 99%

cross-modal fusion techniques for utterance-level emotion recognition from text and speech

Luo¹,

Phan²,

Reiss³

2023

Preprint

View full text Add to dashboard Cite

Multimodal emotion recognition (MER) is a fundamental complex research problem due to the uncertainty of human emotional expression and the heterogeneity gap between different modalities. Audio and text modalities are particularly important for a human participant in understanding emotions. Although many successful attempts have been designed multimodal representations for MER, there still exist multiple challenges to be addressed: 1) bridging the heterogeneity gap between multimodal features and model inter-and intramodal interactions of multiple modalities; 2) effectively and efficiently modeling the contextual dynamics in the conversation sequence. In this paper, we propose Cross-Modal RoBERTa (CM-RoBERTa) model for emotion detection from spoken audio and corresponding transcripts. As the core unit of the CM-RoBERTa, parallel self-and cross-attention is designed to dynamically capture inter-and intra-modal interactions of audio and text. Specially, the mid-level fusion and residual module are employed to model longterm contextual dependencies and learn modality-specific patterns. We evaluate the approach on the MELD dataset and the experimental results show the proposed approach achieves the state-of-art performance on the dataset.

show abstract

“…Most previous studies focus on emotion recognition using only one modality, such as text, video, or voice. However, it has already been shown that algorithms for emotion recognition that are based on multiple modalities, perform better than the ones that use only one modality (Povolny et al, 2016 ; Krishna and Patil, 2020 ). In our study, we go beyond this comparison as we measure the contribution of each encoded modality by intentionally perturbing the input at inference time to either exclude or corrupt some intelligible information, which has already been proven effective for emotion-recognition classification, and monitoring changes in the performance of the model.…”

Section: Introductionmentioning

confidence: 99%

Masking important information to assess the robustness of a multimodal classifier for emotion recognition

Cohen¹,

Rosenberger²,

Butman³

et al. 2023

Front. Artif. Intell.

View full text Add to dashboard Cite

Deep neural networks have been proven effective in classifying human interactions into emotions, especially by encoding multiple input modalities. In this work, we assess the robustness of a transformer-based multimodal audio-text classifier for emotion recognition, by perturbing the input at inference time using attacks which we design specifically to corrupt information deemed important for emotion recognition. To measure the impact of the attacks on the classifier, we compare between the accuracy of the classifier on the perturbed input and on the original, unperturbed input. Our results show that the multimodal classifier is more resilient to perturbation attacks than the equivalent unimodal classifiers, suggesting that the two modalities are encoded in a way that allows the classifier to benefit from one modality even when the other one is slightly damaged.

show abstract

Multimodal Emotion Recognition Using Cross-Modal Attention and 1D Convolutional Neural Networks

Cited by 45 publications

References 13 publications

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

cross-modal fusion techniques for utterance-level emotion recognition from text and speech

Masking important information to assess the robustness of a multimodal classifier for emotion recognition

Contact Info

Product

Resources

About