On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks

Abstract

Speech emotion recognition (SER) is an important aspect of effective human-robot collaboration and received a lot of attention from the research community. For example, many neural network-based architectures were proposed recently and pushed the performance to a new level. However, the applicability of such neural SER models trained only on indomain data to noisy conditions is currently under-researched. In this work, we evaluate the robustness of state-of-the-art neural acoustic emotion recognition models in human-robot interaction scenarios. We hypothesize that a robot's ego noise, room conditions, and various acoustic events that can occur in a home environment can significantly affect the performance of a model. We conduct several experiments on the iCub robot platform and propose several novel ways to reduce the gap between the model's performance during training and testing in realworld conditions. Furthermore, we observe large improvements in the model performance on the robot and demonstrate the necessity of introducing several data augmentation techniques like overlaying background noise and loudness variations to improve the robustness of the neural approaches.

Analysis of Deep Learning Architectures for Cross-Corpus Speech Emotion Recognition

“…SER models trained on a single corpus tend to overfit, leading to poor performance on out-of-domain data, as presented in [3]. To address this issue, several techniques have been proposed: (1) the data augmentation approach, which consists of generating additional training samples by duplicating and often modifying the original training set, using techniques such as vocal tract length perturbation [22] or variation of tempo, loudness and background noise [23]; (2) multi-task learning, in which the models are trained on additional tasks, such as gender or domain identification [24,25]; (3) the transfer learning approach, in which the models are first trained on a given domain and then adapted to the task at hand [26,27]; and (4) cross-modal transfer, in which an image-based emotion recognition model is used to improve SER [28].…”

Incorporating End-to-End Speech Recognition Models for Sentiment Analysis

“…Previous research in end-to-end speech recognition demonstrated the importance of introducing random perturbations into the speech signal like a change of pitch, tempo, loudness, and adding noise [4], [25], [27], [28]. Since such perturbations do not alter the target label (spoken text in the case of speech recognition, or an emotion category), they can be conveniently applied with some occurrence probability during training.…”

“…We only use 4 speakers out of 13 speakers. In our previous work [2], [4] we use the iCub robotic head to test the robustness of our models against the robot's ego noise. However, in this paper, we use the Soundman wooden head to focus on background noise generated by the projectors, computers, air conditioner, power sources as well as noise from airplanes frequently passing nearby, and reverberation noise.…”

“…Also, training on clean texts might result in overstating of the model's performance and degradation during testing when we use even state-of-the-art speech recognition models. This paper is built on our previous work [2], [4] on the application of neural emotion recognition models in the context of human-robot interaction. In this work, we conduct several experiments to address the discrepancy in the performance of a spoken sentiment recognition when the manually transcribed spoken text is not available.…”

Machine Learning Based Emotion Recognition using Speech Signal

“…Here, HRI has utilized as a robotic humanoid head, which resulted in required emotional-feedback. The work [43], [44], [45] presents that the hybrid modality of deep learning might inherit the essential characteristics of RNN through CNN by the levels of convolutional implanted with the RNN. Here, this allows the method for attaining both temporal & frequency dependency in specified speechsignal.…”