Using regional saliency for speech emotion recognition

Aldeneh, Zakaria; Provost, Emily Mower

doi:10.1109/icassp.2017.7952655

Cited by 106 publications

(100 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent times, several neural network approaches have been successfully applied for emotion classification. Researchers have propsoed convolutional neural network (CNN)-based models that are trained on speech utterances for performing identification [14,15,16]. There have been some successful approaches on using attention mechanisms as well [17,18].…”

Section: Recent Workmentioning

confidence: 99%

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Yoon

Dey

Lee

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this work, we explore the impact of visual modality in addition to speech and text for improving the accuracy of the emotion detection system. The traditional approaches tackle this task by fusing the knowledge from the various modalities independently for performing emotion classification. In contrast to these approaches, we tackle the problem by introducing an attention mechanism to combine the information. In this regard, we first apply a neural network to obtain hidden representations of the modalities. Then, the attention mechanism is defined to select and aggregate important parts of the video data by conditioning on the audio and text data. Furthermore, the attention mechanism is again applied to attend important parts of the speech and textual data, by considering other modality. Experiments are performed on the standard IEMOCAP dataset using all three modalities (audio, text, and video). The achieved results show a significant improvement of 3.65% in terms of weighted accuracy compared to the baseline system.

show abstract

Section: Recent Workmentioning

confidence: 99%

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Yoon

Dey

Lee

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Following previous works [10,11], we choose 4 emotion types for our experiments (namely neutral, happy, angry and sad) from the improvised speech for study, since scripted data may contain undesired contextual information. Adopting the methodology of previous works [10,11,23,24], we perform a 10-fold cross-validation using a leave-oneout strategy. In each training process, 9 speakers are used as training data and the remained one is used for testing data.…”

Section: Experiments Settingmentioning

confidence: 99%

“…Furthermore, simply average-pooling or max-pooling may be insufficient to derive effective representations for complex emotional expressions that require analysis of higher order statistics. Some recent works show the benefit of introducing an attention mechanism for representation learning [12,13,10]. However, they generally derive salient regions from the features in a bottomup manner.…”

Section: Introductionmentioning

confidence: 99%

“…The model parameters can be jointly optimized with back-propagation algorithms. In [8,10] a max-pooling operation was applied over time to obtain an utterance representation from salient regions. Neumann el.…”

Section: Introductionmentioning

confidence: 99%

“…Motivated by the success of deep learning techniques in various application domains, such as large scale image and speech recognition [4,5], several Deep Neural Network (DNN) or Convolutional Neural Network (CNN) based SER methods have recently been proposed [6,7,8,9,10,11,12]. In [6,7], a multistage procedure was applied, in which the DNN and CNN network were trained for frontend feature extraction, followed by a backend emotion recognizer such as SVM and Extreme Learning Machine (ELM).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Attention Pooling Based Representation Learning Method for Speech Emotion Recognition

et al. 2018

View full text Add to dashboard Cite

This paper proposes an attention pooling based representation learning method for speech emotion recognition (SER). The emotional representation is learned in an end-to-end fashion by applying a deep convolutional neural network (CNN) directly to spectrograms extracted from speech utterances. Motivated by the success of GoogleNet, two groups of filters with different shapes are designed to capture both temporal and frequency domain context information from the input spectrogram. The learned features are concatenated and fed into the subsequent convolutional layers. To learn the final emotional representation, a novel attention pooling method is further proposed. Compared with the existing pooling methods, such as max-pooling and average-pooling, the proposed attention pooling can effectively incorporate class-agnostic bottom-up, and class-specific top-down, attention maps. We conduct extensive evaluations on benchmark IEMOCAP data to assess the effectiveness of the proposed representation. Results demonstrate a recognition performance of 71.8% weighted accuracy (WA) and 68% unweighted accuracy (UA) over four emotions, which outperforms the state-of-the-art method by about 3% absolute for WA and 4% for UA.

show abstract