Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data

Choi, Woo Yong; Song, Kyu Ye; Lee, Chan Woo

doi:10.18653/v1/w18-3304

Cited by 43 publications

(19 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model for audio-visual person separation can be classified under audio-visual sound separation systems (AV-SS) [6]. Such systems aim to extract one or multiple voice targets from a mixture of voice and visual information [20,21]. This information can then be used in different applications, such as focusing attention [22,23], establishing eye contact [24,25], and building a mutual understanding [26], etc.…”

Section: Current and Comparable Concepts Used For Audio-visual System Perception And Contribution Of Presented Methodologymentioning

confidence: 99%

Person Localization Model Based on a Fusion of Acoustic and Visual Inputs

et al. 2022

View full text Add to dashboard Cite

PLEA is an interactive, biomimetic robotic head with non-verbal communication capabilities. PLEA reasoning is based on a multimodal approach combining video and audio inputs to determine the current emotional state of a person. PLEA expresses emotions using facial expressions generated in real-time, which are projected onto a 3D face surface. In this paper, a more sophisticated computation mechanism is developed and evaluated. The model for audio-visual person separation can locate a talking person in a crowded place by combining input from the ResNet network with input from a hand-crafted algorithm. The first input is used to find human faces in the room, and the second input is used to determine the direction of the sound and to focus attention on a single person. After an information fusion procedure is performed, the face of the person speaking is matched with the corresponding sound direction. As a result of this procedure, the robot could start an interaction with the person based on non-verbal signals. The model was tested and evaluated under laboratory conditions by interaction with users. The results suggest that the methodology can be used efficiently to focus a robot’s attention on a localized person.

show abstract

Section: Current and Comparable Concepts Used For Audio-visual System Perception And Contribution Of Presented Methodologymentioning

confidence: 99%

Person Localization Model Based on a Fusion of Acoustic and Visual Inputs

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Differently, Zadeh et al [43] present Graph-MFN, which synchronizes the multimodal sequences by storing intra-modality and crossmodality interactions through time with a graph structure. Attention mechanism has been exploited by several works as well [44], [45], [41], [46], [47], [15], [17], [48], [21], [49]. For example, Dai et al [21] present MESM that is composed of sparse cross-modal attention mechanism attached to the joint learning of multimodal features.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Franceschini¹,

Fini²,

Beyan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Emotion recognition is involved in several real-world applications. With an increase in available modalities, automatic understanding of emotions is being performed more accurately. The success in Multimodal Emotion Recognition (MER), primarily relies on the supervised learning paradigm. However, data annotation is expensive, time-consuming, and as emotion expression and perception depends on several factors (e.g., age, gender, culture) obtaining labels with a high reliability is hard. Motivated by these, we focus on unsupervised feature learning for MER. We consider discrete emotions, and as modalities text, audio and vision are used. Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature. Our end-to-end feature learning approach has several differences (and advantages) compared to existing MER methods: i) it is unsupervised, so the learning is lack of data labelling cost; ii) it does not require data spatial augmentation, modality alignment, large number of batch size or epochs; iii) it applies data fusion only at inference; and iv) it does not require backbones pre-trained on emotion recognition task. The experiments on benchmark datasets show that our method outperforms several baseline approaches and unsupervised learning methods applied in MER. Particularly, it even surpasses a few supervised MER state-of-the-art.

show abstract

“…The weighted accuracy (WA) and unweighted accuracy (UA) of 64.08 and 56.41% were obtained from the IEMOCAP dataset, respectively. Lee et al (2018) proposed a model combining the convolutional neural network with the attention mechanism and the text data. The promising experimental result in the CMU-MOSEI database proved the effectiveness of the combination of the two modalities.…”

Section: Introductionmentioning

confidence: 99%

Pre-trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition

et al. 2021

View full text Add to dashboard Cite

Speech emotion recognition (SER) is a difficult and challenging task because of the affective variances between different speakers. The performances of SER are extremely reliant on the extracted features from speech signals. To establish an effective features extracting and classification model is still a challenging task. In this paper, we propose a new method for SER based on Deep Convolution Neural Network (DCNN) and Bidirectional Long Short-Term Memory with Attention (BLSTMwA) model (DCNN-BLSTMwA). We first preprocess the speech samples by data enhancement and datasets balancing. Secondly, we extract three-channel of log Mel-spectrograms (static, delta, and delta-delta) as DCNN input. Then the DCNN model pre-trained on ImageNet dataset is applied to generate the segment-level features. We stack these features of a sentence into utterance-level features. Next, we adopt BLSTM to learn the high-level emotional features for temporal summarization, followed by an attention layer which can focus on emotionally relevant features. Finally, the learned high-level emotional features are fed into the Deep Neural Network (DNN) to predict the final emotion. Experiments on EMO-DB and IEMOCAP database obtain the unweighted average recall (UAR) of 87.86 and 68.50%, respectively, which are better than most popular SER methods and demonstrate the effectiveness of our propose method.

show abstract

Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data

Cited by 43 publications

References 10 publications

Person Localization Model Based on a Fusion of Acoustic and Visual Inputs

Person Localization Model Based on a Fusion of Acoustic and Visual Inputs

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Pre-trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition

Contact Info

Product

Resources

About