Emotion recognition in spontaneous and acted dialogues

Abstract

In this work, we compare emotion recognition on two types of speech: spontaneous and acted dialogues. Experiments were conducted on the AVEC2012 database of spontaneous dialogues and the IEMOCAP database of acted dialogues. We studied the performance of two types of acoustic features for emotion recognition: knowledge-inspired disfluency and nonverbal vocalisation (DIS-NV) features, and statistical Low-Level Descriptor (LLD) based features. Both Support Vector Machines (SVM) and Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) were built using each feature set on each emotional database. Our work aims to identify aspects of the data that constrain the effectiveness of models and features. Our results show that the performance of different types of features and models is influenced by the type of dialogue and the amount of training data. Because DIS-NVs are less frequent in acted dialogues than in spontaneous dialogues, the DIS-NV features perform better than the LLD features when recognizing emotions in spontaneous dialogues, but not in acted dialogues. The LSTM-RNN model gives better performance than the SVM model when there is enough training data, but the complex structure of a LSTM-RNN model may limit its performance when there is less training data available, and may also risk over-fitting. Additionally, we find that long distance contexts may be more useful when performing emotion recognition at the word level than at the utterance level.

Variational Autoencoders for Learning Latent Representations of Speech Emotion: A Preliminary Study

“…The overall classification framework has been shown in Figure 1. Previous studies have concluded that the performance of the LSTM model can be enhanced by using more predictive and knowledge-inspired features despite the limited training examples [18,22,23]. Therefore, LSTM is a natural choice for us to use with features generated by VAEs.…”

“…IEMOCAP data were also annotated on three continuous dimensions: Arousal (A), Power (P), and Valence (V). For comparison of our classification results with the state-of-the-art approaches in [18,22], we also consider the above emotion dimensions. However, to maintain it as a classification problem, like [18,22], within each dimension we created three categories: low (values less than 3), mid (values equal to 3) and high (values greater than 3).…”

“…Both VAE-LSTM and CVAE-LSTM significantly outperform the AE-LSTM model, while CVAE-LSTM producing the best performance. Studies [18,22] that we have compared with in Table 2 used different types of features, such as knowledge-inspired disfluency and nonverbal vocalization (DN) features, and statistical Low-Level Descriptor (LLD) features, as an input to the LSTM model. The highest score they achieve is 55.3% (Mean score), which we closely outperform using our proposed CVAE-LSTM model (Mean score 56.42%).…”

Recognizing induced emotions of movie audiences: Are induced and perceived emotions the same?

“…https://www.ibm.com/watson/developercloud/speech-to-text.html (verbal filled pauses), stutters, laughter, and audible breath (remaining words are labelled as general lexicons). DIS-NVs were shown to be indicators of speaker emotions in spontaneous dialogue [10]. To evaluate annotation agreement, we divide the annotations into six subsets based on the DIS-NV labels and compute CC of the word timings in each subset.…”

“…Moreover, cues of perceived emotions in movies may be used for the recognition of induced emotions as well. Thus, we add manual transcripts of the LIRIS-ACCEDE movies, as well as expert annotations of DISfluency and Non-verbal Vocalisations (DIS-NV) in dialogues [10] and aesthetic highlights [11]. In addition, as a comparison with movie based features, we extract physiological and behavioural features based on signals collected from wearable sensors attached to the audience [12].…”