End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks

Abstract

Speech emotion conversion aims to convert the expressed emotion of a spoken utterance to a target emotion while preserving the lexical information and the speaker's identity. In this work, we specifically focus on in-the-wild emotion conversion where parallel data does not exist, and the problem of disentangling lexical, speaker, and emotion information arises. In this paper, we introduce a methodology that uses self-supervised networks to disentangle the lexical, speaker, and emotional content of the utterance, and subsequently uses a HiFiGAN vocoder to resynthesise the disentangled representations to a speech signal of the targeted emotion. For better representation and to achieve emotion intensity control, we specifically focus on the arousal dimension of continuous representations, as opposed to performing emotion conversion on categorical representations. We test our methodology on the large in-the-wild MSP-Podcast dataset. Results reveal that the proposed approach is aptly conditioned on the emotional content of input speech and is capable of synthesising natural-sounding speech for a target emotion. Results further reveal that the methodology better synthesises speech for mid-scale arousal (2 to 6) than for extreme arousal (1 and 7).

End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning

“…This extension is also the first in literature to present SER results in this novel dataset [31]. Existing analyses and experiments from [38], [39] were also extended to MSP-Conversation. Moreover, we performed additional experiments that include an experiment to understand the impact of the number of annotations available, and an ablation study.…”

“…Research efforts have also made to estimate emotion annotations as a distribution, using LDL [26], [38], [39], [47]. Foteinopoulou et al [26] trained a MTL network using a KL divergence loss that models emotion annotations as a uni-variate Gaussian with mean m and unknown variance.…”

“…Chou et al, [47] used LDL to convert perceptual evaluations into histogram-based distributional labels for training. In our preliminary work [38], we modeled emotion annotations as a Gaussian using BBB-based uncertainty modeling. Notwithstanding their improved performances, in [26], [38], [47], a histogram or a Gaussian assumption was made on emotion annotations, despite only having limited annotations.…”

“…This work is based on two prior conference contributions [38], [39], which to the best of our knowledge are the first in the literature to use BBB and LDL in SER. These works were also the first to tackle the problem of limited emotion annotations from an ML perspective.…”

“…These works were also the first to tackle the problem of limited emotion annotations from an ML perspective. While in [38], [39] we only validated the method in one dataset, and with limited experiments, in this extension, we additionally validate the method in a larger and more complex dataset, the MSP-Conversation [31], along with cross-corpora evaluations. This extension is also the first in literature to present SER results in this novel dataset [31].…”