Adapted Dynamic Memory Network for Emotion Recognition in Conversation

Xing, Songlong; Mai, Sijie; Hu, Haifeng

doi:10.1109/taffc.2020.3005660

“…Specifically, Pre-trained 3D-CNN for Human Action [53] is adopted to extract spatio-temporal features of emotions in videos [54] [55]. Furthermore, Pre-trained 3D-CNN for sports [56] are also applied in a series of studies [7] [8] [57] [58] [59] [60]. Resnet-101 [61], as a 3D-CNN architecture pre-trained on the human action video dataset Kinetics [62], is combined with attention mechanism to form the visual models to extract the feature representation of visual stream [63].…”

Section: Preprocessmentioning

confidence: 99%

“…Considering a scene that intelligent robots in customer systems are demanded to recognize emotions of customer after customer speaks in a dialogue, two or more parties existed in the dialogue and a party can be influenced by either its own state in the past or the states of other parties. As a result, a series of novel methods are proposed for ERC [4] [5] [6] [7] [8] [9].…”

Section: Introductionmentioning

confidence: 99%

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Zhang¹,

Tan²

2023

Preprint

3

0

View full text Add to dashboard Cite

With the development of social media and human-computer interaction, it is essential to serve people by perceiving people's emotional state in videos. In recent years, a large number of studies tackle the issue of emotion recognition based on three most common modalities in videos, that is, face, speech and text. The focus of this paper is to sort out the relevant studies of emotion recognition using facial, speech and textual cues based on deep learning techniques due to the lack of review papers concentrating on the three modalities. In this paper, we firstly introduce widely accepted emotion models for the purpose of interpreting the definition of emotion. Then we introduce the state-of-the-art for emotion recognition based on unimodality including facial expression recognition, speech emotion recognition and textual emotion recognition. For multimodal emotion recognition, we summarize the feature-level and decision-level fusion methods in detail. In addition, the description of relevant benchmark datasets, the definition of metrics and the performance of the state-of-the-art in recent years are also outlined for the convenience of readers to find out the current research progress. Ultimately, we explore some potential research challenges and opportunities to give researchers reference for the enrichment of emotion recognition-related researches.

show abstract

“…Specifically, Pre-trained 3D-CNN for Human Action is adopted to extract spatiotemporal features of emotions in videos [52][53]. Furthermore, Pre-trained 3D-CNN for sports are also applied in a series of studies [7] [58], which consists of two stages: Two-Stream convolutional Neural Network and Gated Recurrent Unit Network to capture both micro-and macro-motion, respectively. The feature representation of a snippet, i.e.…”

Section: ) Preprocessmentioning

confidence: 99%

“…[6] propose a model for real-time emotion detection in conversations and experiment in bimodality and multimodality data but lack of the unimodality. [7] and [87] evaluate the performances of their models using different modality combinations but lack the comparison against baselines in unimodality and bimodality. Liang et al [89] evaluate their model and empirical analysis is presented in unimodality, bimodality and mulmodality.…”

Section: Unified Modelmentioning

confidence: 99%

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Zhang¹,

Tan²

2021

Preprint

View full text Add to dashboard Cite

With the development of social media and human-computer interaction, video has become one of the most common data formats. As a research hotspot, emotion recognition system is essential to serve people by perceiving people’s emotional state in videos. In recent years, a large number of studies focus on tackling the issue of emotion recognition based on three most common modalities in videos, that is, face, speech and text. The focus of this paper is to sort out the relevant studies of emotion recognition using facial, speech and textual cues due to the lack of review papers concentrating on the three modalities. On the other hand, because of the effective leverage of deep learning techniques to learn latent representation for emotion recognition, this paper focuses on the emotion recognition method based on deep learning techniques. In this paper, we firstly introduce widely accepted emotion models for the purpose of interpreting the definition of emotion. Then we introduce the state-of-the-art for emotion recognition based on unimodality including facial expression recognition, speech emotion recognition and textual emotion recognition. For multimodal emotion recognition, we summarize the feature-level and decision-level fusion methods in detail. In addition, the description of relevant benchmark datasets, the definition of metrics and the performance of the state-of-the-art in recent years are also outlined for the convenience of readers to find out the current research progress. Ultimately, we explore some potential research challenges and opportunities to give researchers reference for the enrichment of emotion recognition-related researches.

show abstract

“…Considering a scene that intelligent robots in customer systems are demanded to recognize emotions of customer after customer speaks in a dialogue, two or more parties exist in the dialogue and a party can be influenced by either its own state in the past or the states of other parties. As a result, a series of novel methods are proposed for ERC [4][5] [6] [7][8] [9].…”

Section: Introductionmentioning

confidence: 99%

Deep Emotion Recognition in Dynamic Data using Facial, Speech and Textual Cues: A Survey

Zhang¹,

Tan²

2021

Preprint

0

View full text Add to dashboard Cite

With the development of social media and human-computer interaction, video has become one of the most common data formats. As a research hotspot, emotion recognition system is essential to serve people by perceiving people’s emotional state in videos. In recent years, a large number of studies focus on tackling the issue of emotion recognition based on three most common modalities in videos, that is, face, speech and text. The focus of this paper is to sort out the relevant studies of emotion recognition using facial, speech and textual cues due to the lack of review papers concentrating on the three modalities. On the other hand, because of the effective leverage of deep learning techniques to learn latent representation for emotion recognition, this paper focuses on the emotion recognition method based on deep learning techniques. In this paper, we firstly introduce widely accepted emotion models for the purpose of interpreting the definition of emotion. Then we introduce the state-of-the-art for emotion recognition based on unimodality including facial expression recognition, speech emotion recognition and textual emotion recognition. For multimodal emotion recognition, we summarize the feature-level and decision-level fusion methods in detail. In addition, the description of relevant benchmark datasets, the definition of metrics and the performance of the state-of-the-art in recent years are also outlined for the convenience of readers to find out the current research progress. Ultimately, we explore some potential research challenges and opportunities to give researchers reference for the enrichment of emotion recognition-related researches.

show abstract

Adapted Dynamic Memory Network for Emotion Recognition in Conversation

Cited by 47 publications

References 59 publications

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Deep Emotion Recognition in Dynamic Data using Facial, Speech and Textual Cues: A Survey

Contact Info

Product

Resources

About