Discretized Continuous Speech Emotion Recognition with Multi-Task Deep Recurrent Neural Network

Abstract

Estimating continuous emotional states from speech as a function of time has traditionally been framed as a regression problem. In this paper, we present a novel approach that moves the problem into the classification domain by discretizing the training labels at different resolutions. We employ a multi-task deep bidirectional long-short term memory (BLSTM) recurrent neural network (RNN) trained with cost-sensitive Cross Entropy loss to model these labels jointly. We introduce an emotion decoding algorithm that incorporates long-and short-term temporal properties of the signal to produce more robust time series estimates. We show that our proposed approach achieves competitive audio-only performance on the RECOLA dataset, relative to previously published works as well as other strong regression baselines. This work provides a link between regression and classification, and contributes an alternative approach for continuous emotion recognition.

Capturing Long-Term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition

“…We only show results that are reported on the audio modality in the results tables. We also compare our performance to that of an optimized BLSTM regression model, described in [24]. Our final dilated convolution structure has a depth of 10 layers, each having a width of 32.…”

Holistic Affect Recognition Using PaNDA: Paralinguistic Non-Metric Dimensional Analysis

“…A wealth of research pertains to multi-label learning for affect recognition, using a single database labeled with a few affective dimensions, predominantly arousal/ valence/ dominance [12,13,14]. Xia and Liu [15] proposed a method to learn a main task (emotion classification) and a secondary task (arousal/ valence recognition).…”

SeMemNN: A Semantic Matrix-Based Memory Neural Network for Text Classification

“…The most common approach is using a recurrent neural network (RNN) and its variants. Long short term memory (LSTM) has been used in several works to improve the model's ability to catch the long-term dependency in a time series [16,17]. Moreover, For catching the interesting time steps in a sequence, the attention mechanism is also added to the LSTMs and showing its effectiveness [18].…”