Mutual impact of acoustic and linguistic representations for continuous emotion recognition in call-center conversations

Abstract

<div> <div> <div> <p>The goal of our research is to automaticaly retrieve the satisfaction and the frustration in real-life call-center conversations. This study focuses an industrial application in which the customer satisfaction is continuously tracked down to improve customer services. To compensate the lack of large annotated emotional databases, we explore the use of pre-trained speech representations as a form of transfer learning towards AlloSat corpus. Moreover, several studies have pointed out that emotion can be detected not only in speech but also in facial trait, in biological response or in textual information. In the context of telephone conversations, we can break down the audio information into acoustic and linguistic by using the speech signal and its transcription. Our experiments confirms the large gain in performance obtained with the use of pre-trained features. Surprisingly, we found that the linguistic content is clearly the major contributor for the prediction of satisfaction and best generalizes to unseen data. Our experiments conclude to the definitive advantage of using CamemBERT representations, however the benefit of the fusion of acoustic and linguistic modalities is not as obvious. With models learnt on individual annotations, we found that fusion approaches are more robust to the subjectivity of the annotation task. This study also tackles the problem of performances variability and intends to estimate this variability from different views: weights initialization, confidence intervals and annotation subjectivity. A deep analysis on the linguistic content investigates interpretable factors able to explain the high contribution of the linguistic modality for this task. </p> </div> </div> </div>

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

“…Human emotion recognition by machines aids various research fields, such as virtual reality, gaming, robotics, and customer care operations. For instance, if an automated call center system is able to infer a customer's emotional state, the application may be able to provide more appropriate responses or send the call to a human operator straightaway [2]. Emotion detection systems can be used in virtual reality and games to detect a player's emotional suffering, paving the way for more realistic, engaging, and immersive gaming experiences SA 5042, Australia.…”

“…In contrast to automated speech recognition (ASR) datasets, commonly used Speech Emotion Recognition datasets [8]- [10] are restricted in complexity and small in size. Furthermore, systems trained on these datasets may not be generalizable to other domains, such as customer service centers [2]. As a result, self-supervised pre-trained models such as wave-tovector (wav2vec) [11] and Bidirectional Encoder Representations from Transformers (BERT) [12] have been created to address the above-mentioned issue by learning from largescale audio and text datasets without the need of extensive labeling.…”