Speech emotion recognition using convolutional and Recurrent Neural Networks

Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language. In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics. We introduce a novel model, termed Tensor Fusion Network, which learns both such dynamics end-to-end. The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice. In the experiments, our model outperforms state-ofthe-art approaches for both multimodal and unimodal sentiment analysis.

show abstract

“…SER-LSTM (Lim et al, 2016) is a model that uses recurrent neural networks on top of convolution operations on spectrogram of audio.…”

Section: Acoustic Sentiment Analysismentioning

confidence: 99%

Tensor Fusion Network for Multimodal Sentiment Analysis

Zadeh

Chen

Poria

et al. 2017

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

1,050

442

View full text Add to dashboard Cite

show abstract

“…Similarly, Zhang et al [85] used an AlexNet to add increasingly more temporal context to learned feature representations. It is worth noting that while some studies directly used CNN features for affect prediction [142], [128], others combined local temporal modeling with global temporal modeling via RNN [108], [163], or pooling approaches [85].…”

Section: Learning Temporal Features For Sermentioning

confidence: 99%

Deep Learning for Human Affect Recognition: Insights and New Developments

Rouast

Adam

Chiong

2021

IEEE Trans. Affective Comput.

158

View full text Add to dashboard Cite

Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications.

show abstract

“…In [1,2,3] feature learning from raw-waveform or spectrogram using CNN, LSTM based models is explored. In [4,5,6,7], CNN and LSTM based models are explored from feature representations such as MFCC and OpenS-MILE [8] features. In [9,10,11,12], adversarial learning paradigm * Both the authors contributed equally to this paper is explored for robust recognition.…”

Section: Introductionmentioning

confidence: 99%

X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition

Pappagari

Wang

Villalba

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this work, we explore the dependencies between speaker recognition and emotion recognition. We first show that knowledge learned for speaker recognition can be reused for emotion recognition through transfer learning. Then, we show the effect of emotion on speaker recognition. For emotion recognition, we show that using a simple linear model is enough to obtain good performance on the features extracted from pre-trained models such as the x-vector model. Then, we improve emotion recognition performance by finetuning for emotion classification. We evaluated our experiments on three different types of datasets: IEMOCAP, MSP-Podcast, and Crema-D. By fine-tuning, we obtained 30.40%, 7.99%, and 8.61% absolute improvement on IEMOCAP, MSP-Podcast, and Crema-D respectively over baseline model with no pre-training. Finally, we present results on the effect of emotion on speaker verification. We observed that speaker verification performance is prone to changes in test speaker emotions. We found that trials with angry utterances performed worst in all three datasets. We hope our analysis will initiate a new line of research in the speaker recognition community.

show abstract

Speech emotion recognition using convolutional and Recurrent Neural Networks

Cited by 306 publications

References 8 publications

Tensor Fusion Network for Multimodal Sentiment Analysis

Tensor Fusion Network for Multimodal Sentiment Analysis

Deep Learning for Human Affect Recognition: Insights and New Developments

X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition

Contact Info

Product

Resources

About