Real-Time Speech Emotion and Sentiment Recognition for Interactive Dialogue Systems

Abstract

In this paper, we describe our approach of enabling an interactive dialogue system to recognize user emotion and sentiment in realtime. These modules allow otherwise conventional dialogue systems to have "empathy" and answer to the user while being aware of their emotion and intent. Emotion recognition from speech previously consists of feature engineering and machine learning where the first stage causes delay in decoding time. We describe a CNN model to extract emotion from raw speech input without feature engineering. This approach even achieves an impressive average of 65.7% accuracy on six emotion categories, a 4.5% improvement when compared to the conventional feature based SVM classification. A separate, CNN-based sentiment analysis module recognizes sentiments from speech recognition results, with 82.5 Fmeasure on human-machine dialogues when trained with out-of-domain data.

MoEL: Mixture of Empathetic Listeners

“…So far, to solve the problem of empathetic dialogue response generation, which is to understand the user emotion and respond appropriately (Bertero et al, 2016), there have been mainly two lines arXiv:1908.07687v1 [cs.CL] 21 Aug 2019 of work. The first is a multi-task approach that jointly trains a model to predict the current emotional state of the user and generate an appropriate response based on the state (Lubis et al, 2018;Rashkin et al, 2018).…”

Self-Attention for Speech Emotion Recognition

“…For this reason, we want to create a fully end-to-end model that we hope that will be able to automatically extract meaningful features and compare it with a standard features extraction method. [14], [15] and [16] are only a few of the publications that show the effectiveness of convolution as features extraction layers from raw audio. A drawback of this approach is that by adding the features extraction task to the model pipeline additional complexity is added to the end to end system increasing the need in the number of datapoints for training a robust model.…”

CAiRE_HKUST at SemEval-2019 Task 3: Hierarchical Attention for Dialogue Emotion Classification

“…Also, other models, such as a deep averaging network (Iyyer et al, 2015), attention-based network (Winata et al, 2018), and memory network (Dou, 2017), have been investigated to improve the classification performance. Practically, the application of emotion classification has been investigated on interactive dialogue systems (Bertero et al, 2016;Winata et al, 2017;Siddique et al, 2017;.…”