Robust Speech Emotion Recognition Under Different Encoding Conditions

Abstract

In an era where large speech corpora annotated for emotion are hard to come by, and especially ones where emotion is expressed freely instead of being acted, the importance of using free online sources for collecting such data cannot be overstated. Most of those sources, however, contain encoded audio due to storage and bandwidth constraints, often in very low bitrates. In addition, with the increased industry interest on voice-based applications, it is inevitable that speech emotion recognition (SER) algorithms will soon find their way into production environments, where the audio might be encoded in a different bitrate than the one available during training. Our contribution is threefold. First, we show that encoded audio still contains enough relevant information for robust SER. Next, we investigate the effects of mismatched encoding conditions in the training and test set both for traditional machine learning algorithms built on hand-crafted features and modern end-toend methods. Finally, we investigate the robustness of those algorithms in the multi-condition scenario, where the training set is augmented with encoded audio, but still differs from the training set. Our results indicate that end-to-end methods are more robust even in the more challenging scenario of mismatched conditions.

Dawn of the transformer era in speech emotion recognition: closing the valence gap

“…Although this field has seen tremendous progress in the last decades [1], three major challenges remain for real-world paralinguistics-based SER applications: a) improving on its inferior valence performance [4,8], b) overcoming issues of generalisation and robustness [12,13], and c) alleviating individual-and group-level fairness concerns, which is a prerequisite for ethical emotion recognition technology [14,15]. Previous works have attempted to tackle these issues in isolation, e. g. by using cross-modal knowledge distillation to increase valence performance [16], speech enhancement or data augmentation to improve robustness [12,13], and de-biasing techniques to mitigate unfair outcomes [17]. However, each of those approaches comes with its own knobs to twist and hyperparameters to tune, making their combination far from straightforward.…”

Probing Speech Emotion Recognition Transformers for Linguistic Knowledge

“…In particular, this lets us investigate whether the fine-tuning is necessary for adapting to acoustic mismatches between the pre-training and downstream domains, as previously shown for convolutional neural networks (CNNs) [19], or to better leverage linguistic information. This type of behavioural testing goes beyond past work that typically investigates SER models' robustness with respect to noise and small perturbations [20,21,22] or fairness [23,24], thus, providing better insights into the inner workings of SER models.…”