Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation

Padi, Sarala; Sadjadi, Seyed Omid; Sriram, Ram D.; Manocha, Dinesh

doi:10.1145/3462244.3481003

Cited by 39 publications

(15 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides investigating performance of SER models on clean test data, it is important to show that they also work well under more challenging conditions. Even though augmentation methods have been used to improve performance on clean test data [36,37], only a few studies have evaluated performance on augmented test data as well. Jaiswal and Provost [38] and Pappagari et al [39] have shown that previous SER models show robustness issues, particularly for background noise and reverb.…”

mentioning

confidence: 99%

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Wagner¹,

Triantafyllopoulos²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks. In the audio domain, such architectures have also been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of .638 on MSP-Podcast. Furthermore, our investigations reveal that transformer-based architectures are more robust to small perturbations compared to a CNN-based baseline and fair with respect to biological sex groups, but not towards individual speakers. Finally, we are the first to show that their extraordinary success on valence is based on implicit linguistic information learnt during finetuning of the transformer layers, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. Our findings collectively paint the following picture: transformer-based architectures constitute the new state-of-the-art in SER, but further advances are needed to mitigate remaining robustness and individual speaker issues. To make our findings reproducible, we release the best performing model to the community.

show abstract

mentioning

confidence: 99%

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Wagner¹,

Triantafyllopoulos²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To further distinguish between important and non-speech parts of the input, an attention mechanism, similar to [29], was used before the classifier. Apart from the attention-based approaches, transfer learning [33] and various augmentation techniques [25] have been developed to deal with the limited amount of available natural speech data. None of the existing SER models considers the users' privacy challenge, which can significantly affect their applicability in real-life applications.…”

Section: Related Workmentioning

confidence: 99%

“…Since the improvised corpus is closer to natural speech and can elicit more intense emotions, we use only the improvised raw audio samples from the dataset. Additionally, as most papers on SER have targeted the improvised corpus, with a focus on the detection of four core emotion -Neutral, Happy, Sad and Angry [28], [32], [33], we use these 4 emotions to be able to compare our results. Simulation Environment and Setup: To simulate a federated environment, we use the Flower framework [37] and utilize FedAvg [15] as an optimization algorithm to construct the global model from devices' local updates.…”

Section: Datasetsmentioning

confidence: 99%

Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning

Tsouvalas¹,

Özçelebi²,

Meratnia³

2022

Preprint

View full text Add to dashboard Cite

Speech Emotion Recognition (SER) refers to the recognition of human emotions from natural speech. If done accurately, it can offer a number of benefits in building humancentered context-aware intelligent systems. Existing SER approaches are largely centralized, without considering users' privacy. Federated Learning (FL) is a distributed machine learning paradigm dealing with decentralization of privacy-sensitive personal data. In this paper, we present a privacy-preserving and data-efficient SER approach by utilizing the concept of FL. To the best of our knowledge, this is the first federated SER approach, which utilizes self-training learning in conjunction with federated learning to exploit both labeled and unlabeled on-device data. Our experimental evaluations on the IEMOCAP dataset shows that our federated approach can learn generalizable SER models even under low availability of data labels and highly non-i.i.d. distributions. We show that our approach with as few as 10% labeled data, on average, can improve the recognition rate by 8.67% compared to the fully-supervised federated counterparts.

show abstract

“…Currently, we see the massive application of deep learning (DL) in different fields, including SER. DL techniques that are used to improve SER performance may include Deep Neural Networks (DNN) of various architectures [26], Generative Adversarial Networks (GAN) [44,45], autoencoders [46,47], Extreme Learning Machines (ELM) [48], multitask learning [49], transfer learning [50], attention mechanisms [26], etc.…”

Section: Classifiersmentioning

confidence: 99%

Automatic Speech Emotion Recognition of Younger School Age Children

Matveev¹,

Matveev²,

Frolova³

et al. 2022

Mathematics

View full text Add to dashboard Cite

This paper introduces the extended description of a database that contains emotional speech in the Russian language of younger school age (8–12-year-old) children and describes the results of validation of the database based on classical machine learning algorithms, such as Support Vector Machine (SVM) and Multi-Layer Perceptron (MLP). The validation is performed using standard procedures and scenarios of the validation similar to other well-known databases of children’s emotional acting speech. Performance evaluation of automatic multiclass recognition on four emotion classes “Neutral (Calm)—Joy—Sadness—Anger” shows the superiority of SVM performance and also MLP performance over the results of perceptual tests. Moreover, the results of automatic recognition on the test dataset which was used in the perceptual test are even better. These results prove that emotions in the database can be reliably recognized both by experts and automatically using classical machine learning algorithms such as SVM and MLP, which can be used as baselines for comparing emotion recognition systems based on more sophisticated modern machine learning methods and deep neural networks. The results also confirm that this database can be a valuable resource for researchers studying affective reactions in speech communication during child-computer interactions in the Russian language and can be used to develop various edutainment, health care, etc. applications.

show abstract

Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation

Cited by 39 publications

References 50 publications

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning

Automatic Speech Emotion Recognition of Younger School Age Children

Contact Info

Product

Resources

About