Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation

Xu, Mingke; Zhang, Fan; Cui, Xiaodong; Zhang, Wei

doi:10.1109/icassp39728.2021.9414635

Cited by 52 publications

(24 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides investigating performance of SER models on clean test data, it is important to show that they also work well under more challenging conditions. Even though augmentation methods have been used to improve performance on clean test data [36,37], only a few studies have evaluated performance on augmented test data as well. Jaiswal and Provost [38] and Pappagari et al [39] have shown that previous SER models show robustness issues, particularly for background noise and reverb.…”

mentioning

confidence: 99%

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Wagner¹,

Triantafyllopoulos²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks. In the audio domain, such architectures have also been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of .638 on MSP-Podcast. Furthermore, our investigations reveal that transformer-based architectures are more robust to small perturbations compared to a CNN-based baseline and fair with respect to biological sex groups, but not towards individual speakers. Finally, we are the first to show that their extraordinary success on valence is based on implicit linguistic information learnt during finetuning of the transformer layers, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. Our findings collectively paint the following picture: transformer-based architectures constitute the new state-of-the-art in SER, but further advances are needed to mitigate remaining robustness and individual speaker issues. To make our findings reproducible, we release the best performing model to the community.

show abstract

mentioning

confidence: 99%

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Wagner¹,

Triantafyllopoulos²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To compare FrAUG with conventional data augmentation techniques, DepAudioNet models were trained using the DAIC-WOZ dataset with FrAUG, noise augmentation [25], VTLP-based augmentation [32,33], speed perturbation and pitch perturbation [16]. The noise augmentation method was similar to the one used in Kaldi.…”

Section: Fraug Versus Conventional Data Augmentation Methodsmentioning

confidence: 99%

FrAUG: A Frame Rate Based Data Augmentation Method for Depression Detection from Speech Signals

Ravi¹,

Wang²,

Flint³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, a data augmentation method is proposed for depression detection from speech signals. Samples for data augmentation were created by changing the frame-width and the frame-shift parameters during the feature extraction process. Unlike other data augmentation methods (such as VTLP, pitch perturbation, or speed perturbation), the proposed method does not explicitly change acoustic parameters but rather the time-frequency resolution of frame-level features. The proposed method was evaluated using two different datasets, models, and input acoustic features. For the DAIC-WOZ (English) dataset when using the DepAudioNet model and mel-Spectrograms as input, the proposed method resulted in an improvement of 5.97% (validation) and 25.13% (test) when compared to the baseline. The improvements for the CON-VERGE (Mandarin) dataset when using the x-vector embeddings with CNN as the backend and MFCCs as input features were 9.32% (validation) and 12.99% (test). Baseline systems do not incorporate any data augmentation. Further, the proposed method outperformed commonly used data-augmentation methods such as noise augmentation, VTLP, Speed, and Pitch Perturbation. All improvements were statistically significant.

show abstract

“…Furthermore, a global-aware fusion module is introduced to select important information reflecting emotional patterns in different scales. Experiments show both multi-scale feature representation and global-aware fusion can bring great improvements compared to the state-of-the-art CNN model [9].…”

Section: Introductionmentioning

confidence: 99%

“…Recently, inspired by the uplifting progress in computer vision, existing studies [2,3,4,5,6,7,8,9] have achieved great improvements on SER by viewing spectral features as images. The standard convolutional neural network (CNN) architecture mainly consists of feature representation and attention map.…”

Section: Introductionmentioning

confidence: 99%

“…Generally, feature representation usually learns fixed-scale features for capturing all informative features. Besides, attention map is utilized to attend most emotional information among local fixed-scale features [6,7,8,9]. However, emotions are characterized by articulation variations with different scales in time-varied spectral feature.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Speech Emotion Recognition with Global-Aware Fusion on Multi-Scale Feature Representation

Zhu

2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Speech Emotion Recognition (SER) is a fundamental task to predict the emotion label from speech data. Recent works mostly focus on using convolutional neural networks (CNNs) to learn local attention map on fixed-scale feature representation by viewing time-varied spectral features as images. However, rich emotional feature at different scales and important global information are not able to be well captured due to the limits of existing CNNs for SER. In this paper, we propose a novel GLobal-Aware Multi-scale (GLAM) neural network 1 to learn multi-scale feature representation with global-aware fusion module to attend emotional information. Specifically, GLAM iteratively utilizes multiple convolutional kernels with different scales to learn multiple feature representation. Then, instead of using attention-based methods, a simple but effective global-aware fusion module is applied to grab most important emotional information globally. Experiments on the benchmark corpus IEMOCAP over four emotions demonstrates the superiority of our proposed model with 2.5% to 4.5% improvements on four common metrics compared to previous state-of-the-art approaches.

show abstract

Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation

Cited by 52 publications

References 20 publications

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Dawn of the transformer era in speech emotion recognition: closing the valence gap

FrAUG: A Frame Rate Based Data Augmentation Method for Depression Detection from Speech Signals

Speech Emotion Recognition with Global-Aware Fusion on Multi-Scale Feature Representation

Contact Info

Product

Resources

About