Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognition

Abstract

Traditionally, in paralinguistic analysis for emotion detection from speech, emotions have been identified with discrete or dimensional (continuous-valued) labels. Accordingly, models that have been proposed for emotion detection use one or the other of these label types. However, psychologists like Russell and Plutchik have proposed theories and models that unite these views, maintaining that these representations have shared and complementary information. This paper is an attempt to validate these viewpoints computationally. To this end, we propose a model to jointly predict continuous and discrete emotional attributes and show how the relationship between these can be utilized to improve the robustness and performance of emotion recognition tasks. Our approach comprises multi-task and hierarchical multi-task learning frameworks that jointly model the relationships between continuous-valued and discrete emotion labels. Experimental results on two widely used datasets (IEMOCAP and MSPPodcast) for speech-based emotion recognition show that our model results in statistically significant improvements in performance over strong baselines with non-unified approaches. We also demonstrate that using one type of label (discrete or continuousvalued) for training improves recognition performance in tasks that use the other type of label. Experimental results and reasoning for this approach (called mis-matched training approach) are also presented.

Speech Emotion Classification from Affective Dimensions: Limitation and Advantage

“…Although this study has shown the limitation of this approach with a state-of-the-art classifier/regressor, the benefits from dimensional annotation for classification are not new in the literature. For example, [18]- [20] suggested employing both annotations in a multitask approach to improve classification results. Although these studies showed that dimensional information can be helpful for classification, our study has indicated the power of only dimensional information in the combined paradigm.…”