ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection

Hazarika, Devamanyu; Poria, Soujanya; Mihalcea, Rada; Cambria, Erik; Zimmermann, Roger

doi:10.18653/v1/d18-1280

Cited by 329 publications

(198 citation statements)

References 38 publications

Supporting

Mentioning

198

Contrasting

Order By: Relevance

“…Recent works on ERC, e.g., DialogueRNN [11] or ICON [23], strive to address several key research challenges that make the task of ERC difficult to solve: a) Categorization of emotions: Emotion is defined using two type of models -categorical and dimensional. Categorical model classifies emotion into a fixed number of discrete categories.…”

Section: Research Challengesmentioning

confidence: 99%

“…Due to the sequential nature of the utterances in conversations, RNNs are used for context generation in the aforemen- utterances train val test train val test IEMOCAP 120 31 5810 1623 SEMAINE 63 32 4368 1430 EmotionLines 720 80 200 10561 1178 2764 MELD 1039 114 280 9989 1109 2610 [23] as in our experiment, we disregard their contextual feature extraction and pre-processing part. and More details can be found in Majumder et al [11].…”

Section: Recent Advancesmentioning

confidence: 99%

See 1 more Smart Citation

Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances

Majumder²,

et al. 2019

Self Cite

View full text Add to dashboard Cite

Emotion is intrinsic to humans and consequently emotion understanding is a key part of human-like artificial intelligence (AI). Emotion recognition in conversation (ERC) is becoming increasingly popular as a new research frontier in natural language processing (NLP) due to its ability to mine opinions from the plethora of publicly available conversational data in platforms such as Facebook, Youtube, Reddit, Twitter, and others. Moreover, it has potential applications in health-care systems (as a tool for psychological analysis), education (understanding student frustration) and more. Additionally, ERC is also extremely important for generating emotion-aware dialogues that require an understanding of the user's emotions. Catering to these needs calls for effective and scalable conversational emotion-recognition algorithms. However, it is a strenuous problem to solve because of several research challenges. In this paper, we discuss these challenges and shed light on the recent research in this field. We also describe the drawbacks of these approaches and discuss the reasons why they fail to successfully overcome the research challenges in ERC.

show abstract

Section: Research Challengesmentioning

confidence: 99%

Section: Recent Advancesmentioning

confidence: 99%

Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances

Majumder²,

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Various approaches based on deep [7], convolutional [10] and recurrent neural networks [3] have been proposed for speech and video based emotion recognition and text-based sentiment analysis. Variants of such networks with memory blocks [11] and attention mechanisms [12] were developed recently for multi-modal emotion recognition.…”

Section: Introductionmentioning

confidence: 99%

“…Rather, such models are useful in modeling complex inter-modality dynamics. ICON model [11] uses early fusion and dialog-level decision making by considering the histories of the dyadicinteractions. However, such systems require long contextual information which is not readily available in typical real-time interactions between humans and computer.…”

Section: Introductionmentioning

confidence: 99%

Fusion Techniques for Utterance-Level Emotion Recognition Combining Speech and Transcripts

Sebastian

Pierucci²

2019

Interspeech 2019

View full text Add to dashboard Cite

In human perception and understanding, a number of different and complementary cues are adopted according to different modalities. Various emotional states in communication between humans reflect this variety of cues across modalities. Recent developments in multi-modal emotion recognition utilize deeplearning techniques to achieve remarkable performances, with models based on different features suitable for text, audio and vision. This work focuses on cross-modal fusion techniques over deep learning models for emotion detection from spoken audio and corresponding transcripts. We investigate the use of long short-term memory (LSTM) recurrent neural network (RNN) with pre-trained word embedding for text-based emotion recognition and convolutional neural network (CNN) with utterance-level descriptors for emotion recognition from speech. Various fusion strategies are adopted on these models to yield an overall score for each of the emotional categories. Intra-modality dynamics for each emotion is captured in the neural network designed for the specific modality. Fusion techniques are employed to obtain the inter-modality dynamics. Speaker and session-independent experiments on IEMOCAP multi-modal emotion detection dataset show the effectiveness of the proposed approaches. This method yields state-of-the-art results for utterance-level emotion recognition based on speech and text.

show abstract

“…Although, researchers mainly focus on emotion detection on text in absence of context (Klinger et al, 2018), tipically extracted from social media, recently, there are few works that approach the emotion detection in conversations by using context information (Hazarika et al, 2018b) (Majumder et al, 2018) (Hazarika et al, 2018a). These contextual systems work on long conversations where different users are involved and they use multimodal data, specifically, text, audio and video in order to address the emotion detection problem on large multi-party conversations.…”

Section: Introductionmentioning

confidence: 99%

ELiRF-UPV at SemEval-2019 Task 3: Snapshot Ensemble of Hierarchical Convolutional Neural Networks for Contextual Emotion Detection

González

Hurtado

Pla

2019

Proceedings of the 13th International Workshop on Semantic Evaluation

View full text Add to dashboard Cite

This paper describes the approach developed by the ELiRF-UPV team at SemEval 2019 Task 3: Contextual Emotion Detection in Text. We have developed a Snapshot Ensemble of 1D Hierarchical Convolutional Neural Networks to extract features from 3-turn conversations in order to perform contextual emotion detection in text. This Snapshot Ensemble is obtained by averaging the models selected by a Genetic Algorithm that optimizes the evaluation measure. The proposed ensemble obtains better results than a single model and it obtains competitive and promising results on Contextual Emotion Detection in Text.

show abstract

ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection

Cited by 329 publications

References 38 publications

Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances

Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances

Fusion Techniques for Utterance-Level Emotion Recognition Combining Speech and Transcripts

ELiRF-UPV at SemEval-2019 Task 3: Snapshot Ensemble of Hierarchical Convolutional Neural Networks for Contextual Emotion Detection

Contact Info

Product

Resources

About