Multi-Modal Learning for Speech Emotion Recognition: An Analysis and Comparison of ASR Outputs with Ground Truth Transcription

Abstract

In this paper we plan to leverage multi-modal learning and automated speech recognition (ASR) systems toward building a speech-only emotion recognition model. Previous studies have shown that emotion recognition models using only acoustic features do not perform satisfactorily in detecting valence level. Text analysis has been shown to be helpful for sentiment classification. We compared classification accuracies obtained from an audio-only model, a text-only model and a multi-modal system leveraging both by performing a cross-validation analysis on IEMOCAP dataset. Confusion matrices show it's the valence level detection thats being improved by incorporating textual information. In the second stage of experiments, we used two ASR application programming interfaces (APIs) to get the transcriptions. We compare the performances of multi-modal systems using the ASR transcriptions with each other and with that of one using ground truth transcription. We analyze the confusion matrices to determine the effect of using ASR transcriptions instead of ground truth ones on class-wise accuracies. We investigate the generalisability of such a model by performing a cross-corpus study.

Dimensional speech emotion recognition from speech features and word embeddings by using multitask learning

“…Feature set/modality, which attained the highest performance in the categorical approach, is different from the dimensional approach. In the categorical approach with the IEMOCAP dataset, word embeddings gave the highest performance in the unimodal model, as reported in [1,2,4,25]. In contrast, in the dimensional approach, the average performance of acoustic features gave better performance over text features.…”

“…In [1,23], the authors used different deep learning architectures to predict categorical emotion from both speech and text. Some authors used phonemes instead of text for predicting emotion category, such as in [3,24] and another author compared text feature from ASR with manual transcription to investigate the effectiveness of its combination with acoustic features for categorical emotion recognition [25]. Those research, although used audio and text features, only predicted categorical emotion.…”

“…Although pAA was designed for multipurpose audio analysis, some research has reported its effectiveness for affective speech research, including speech emotion recognition and depression classification (e.g. [25]).…”

Deep Architecture Enhancing Robustness to Noise, Adversarial Attacks, and Cross-Corpus Setting for Speech Emotion Recognition

“…We observe that the proposed model achieves better performance and data augmentation helps to improve the robustness. We also compare our results with previous studies ( [36,41]) in the cross-corpus setting in Table 4. In [36], authors employ a multi-task framework and exploit larger unlabelled data for the auxiliary task to improve the generalisation of the model.…”

“…In [36], authors employ a multi-task framework and exploit larger unlabelled data for the auxiliary task to improve the generalisation of the model. In [41], the authors develop a multi-modal technique (audio plus text) for SER based on ASR transcriptions. They demonstrate that the generalisability of ASR models helps to improve the generalisation of emotion classification models.…”