Multitask Learning From Augmented Auxiliary Data for Improving Speech Emotion Recognition

Latif, Siddique; Rana, Rajib; Khalifa, Sara; Jurdak, Raja; Schuller, Björn

doi:10.1109/taffc.2022.3221749

Cited by 14 publications

(6 citation statements)

References 98 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We chose it for the following reasons. Firstly, this database is often used in research on emotional speech [22,23] and is widely available. Secondly, as presented by the authors of [24], this database enables the highest efficiency of classification (88.47% [25]) compared to other databases, such as RAVDESS (87.5% [15,26]) or IEMOCAP (75.60% [27,28]).…”

Section: Audio Datamentioning

confidence: 99%

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Smietanka,

Maka

2023

Journal of Artificial Intelligence and Soft Computing Research

View full text Add to dashboard Cite

The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the Grad-CAM algorithm, energy distribution in the time–frequency space and their relationship with prosodic properties of the considered emotional utterances have been analysed. After preliminary experiments with the expressive speech classification task, we have selected the CQT-96 time–frequency representation. Also, we have used a custom CNN architecture with three convolutional layers in the main experimental phase of the study. Based on the performed analysis, we show the relationship between activation levels and changes in the voiced parts of the fundamental frequency trajectories. As a result, the relationships between the individual activation maps, energy distribution, and fundamental frequency trajectories for six emotional states were described. The results show that the convolutional neural network in the learning process uses similar fragments from time–frequency representation, which are also related to the prosodic properties of emotional speech utterances. We also analysed the relations of the obtained activation maps with time-domain envelopes. It allowed observing the importance of the speech signals energy in classifying individual emotional states. Finally, we compared the energy distribution of the CQT representation in relation to the regions’ energy overlapping with masks of individual emotional states. In the result, we obtained information on the variability of energy distributions in the selected signal representation speech for particular emotions.

show abstract

Section: Audio Datamentioning

confidence: 99%

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Smietanka,

Maka

2023

Journal of Artificial Intelligence and Soft Computing Research

View full text Add to dashboard Cite

show abstract

“…Te hierarchical multitask learning framework is also proposed that takes the coarse classifcation and fne classifcation as two tasks [28]. Te augmentation of data and unsupervised reconstruction can be taken as an auxiliary task to avoid the difculties caused by the data annotation [29]. Another method is more complicated that obtains the multiscale unifed metric [30] by the multitask learning, where the classifcation of both Emission States Category and Emission Intensity Scale is the main task and the classifcation of phone recognition and gender recognition is the auxiliary task.…”

Section: Multitask Learningmentioning

confidence: 99%

“…Te hierarchical multitask learning is proposed that uses the coarse classifcation and fne classifcation as two tasks [28]. Tere methods use unsupervised reconstruction as an auxiliary task [29]. Te more complicated method obtains the multiscale unifed metric [30], where the phone recognition and gender recognition are the auxiliary tasks.…”

Section: Introductionmentioning

confidence: 99%

Multimodal and Multitask Learning with Additive Angular Penalty Focus Loss for Speech Emotion Recognition

Wen,

Ye,

et al. 2023

International Journal of Intelligent Systems

View full text Add to dashboard Cite

Speech emotion recognition has lots of applications such as human-computer interaction and health management. The current methods are challenged with the problems of fuzzy decision boundary and imbalance between difficult and easy samples in the training data. This paper first proposes an additive angle penalty focus loss function (APFL), which strictly refines the fuzzy decision boundary by introducing angle penalty factors to improve the compactness within the class and enlarge the distance between classes. It also assigns the larger loss to difficult samples to make the model pay more attention to them, as they are easily misclassified. Simultaneously, due to the lack of training samples, the framework of multimodal and multitask learning with APFL is further proposed, which extracts spectrogram features by deep neural network, text features by the pretrained language model, and audio features by the pretrained sound model. It uses the gender recognition as an auxiliary task. The experimental results verify the effectiveness of the proposed loss function and framework.

show abstract

“…To increase the size of the training data when the construction of a large-scale emotional speech corpus is difficult, several approaches [7][8][9][10][11][12][13] have used multiple emotional speech corpora in training. Another class of approaches tries to enhance the generalization capability by introducing a variety of regularization approaches and metric losses [7,11,[14][15][16][17][18][19][20][21][22][23].…”

Section: Introductionmentioning

confidence: 99%

Speech Emotion Recognition Incorporating Relative Difficulty and Labeling Reliability

Ahn,

Han,

Lee

et al. 2024

Sensors

View full text Add to dashboard Cite

Emotions in speech are expressed in various ways, and the speech emotion recognition (SER) model may perform poorly on unseen corpora that contain different emotional factors from those expressed in training databases. To construct an SER model robust to unseen corpora, regularization approaches or metric losses have been studied. In this paper, we propose an SER method that incorporates relative difficulty and labeling reliability of each training sample. Inspired by the Proxy-Anchor loss, we propose a novel loss function which gives higher gradients to the samples for which the emotion labels are more difficult to estimate among those in the given minibatch. Since the annotators may label the emotion based on the emotional expression which resides in the conversational context or other modality but is not apparent in the given speech utterance, some of the emotional labels may not be reliable and these unreliable labels may affect the proposed loss function more severely. In this regard, we propose to apply label smoothing for the samples misclassified by a pre-trained SER model. Experimental results showed that the performance of the SER on unseen corpora was improved by adopting the proposed loss function with label smoothing on the misclassified data.

show abstract

Multitask Learning From Augmented Auxiliary Data for Improving Speech Emotion Recognition

Cited by 14 publications

References 98 publications

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Multimodal and Multitask Learning with Additive Angular Penalty Focus Loss for Speech Emotion Recognition

Speech Emotion Recognition Incorporating Relative Difficulty and Labeling Reliability

Contact Info

Product

Resources

About