Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis

Poria, Soujanya; Wang, Zhaoxia; Hazarika, Devamanyu; Mazumder, Navonil; Zadeh, Amir; Morency, Louis–Philippe

doi:10.1109/icdm.2017.134

Cited by 146 publications

(63 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We compare HFFN with following multimodal algorithms: RMFN (Liang et al, 2018a), MFN (Zadeh et al, 2018a), MCTN (Pham et al, 2019), BC-LSTM (Poria et al, 2017b), TFN , MARN (Zadeh et al, 2018b), LMF ), MFM (Tsai et al, 2019, MR-RF (Barezi et al, 2018), FAF (Gu et al, 2018b), RAVEN (Wang et al, 2019), GMFN (Zadeh et al, 2018c), Memn2n (Sukhbaatar et al, 2015), MM-B2 , CHFusion (Majumder et al, 2018), SVM Trees (Rozgic et al, 2012), CMN , C-MKL (Poria et al, 2016b) and CAT-LSTM (Poria et al, 2017c).…”

Section: Comparison With Baselinesmentioning

confidence: 99%

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Mai¹,

Hu²,

Xing³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

104

View full text Add to dashboard Cite

We propose a general strategy named 'divide, conquer and combine' for multimodal fusion. Instead of directly fusing features at holistic level, we conduct fusion hierarchically so that both local and global interactions are considered for a comprehensive interpretation of multimodal embeddings. In the 'divide' and 'conquer' stages, we conduct local fusion by exploring the interaction of a portion of the aligned feature vectors across various modalities lying within a sliding window, which ensures that each part of multimodal embeddings are explored sufficiently. On its basis, global fusion is conducted in the 'combine' stage to explore the interconnection across local interactions, via an Attentive Bi-directional Skipconnected LSTM that directly connects distant local interactions and integrates two levels of attention mechanism. In this way, local interactions can exchange information sufficiently and thus obtain an overall view of multimodal information. Our method achieves state-ofthe-art performance on multimodal affective computing with higher efficiency.

show abstract

Section: Comparison With Baselinesmentioning

confidence: 99%

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Mai¹,

Hu²,

Xing³

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

104

View full text Add to dashboard Cite

show abstract

“…Literature consists of numerous fusion techniques for multimodal data (Atrey et al, 2010;Zadeh et al, 2017;Poria et al, 2017c). Exploring these on CMN, however, is beyond the scope of this paper and left as a future work.…”

Section: Fusionmentioning

confidence: 99%

Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos

Hazarika¹,

Poria²,

Zadeh³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

Self Cite

344

219

View full text Add to dashboard Cite

Emotion recognition in conversations is crucial for the development of empathetic machines. Present methods mostly ignore the role of inter-speaker dependency relations while classifying emotions in conversations. In this paper, we address recognizing utterance-level emotions in dyadic conversational videos. We propose a deep neural framework, termed conversational memory network, which leverages contextual information from the conversation history. The framework takes a multimodal approach comprising audio, visual and textual features with gated recurrent units to model past utterances of each speaker into memories. Such memories are then merged using attention-based hops to capture inter-speaker dependencies. Experiments show an accuracy improvement of 3−4% over the state of the art.

show abstract

“…Advancing from the existing data fusion method, various data fusion methodologies have been proposed by applying attention mechanisms rather than simply focusing on various inter-modality dynamics through various combinations of data fusion. For example, CAT-LSTM improves performance by applying an attention module when performing data fusion or utterance classification [17]. A study to improve the performance of multimodal sentiment analysis by using contextual inter-modal attention has been proposed [18].…”

Section: A Multimodal Sentiment Analysismentioning

confidence: 99%

“…The evaluation was conducted by dividing the training, validation, and test sets according to the split criteria used in previous research [17], [18]. In the experiment using the CMU-MOSI dataset, we evaluated for binary classification of the 'positive' and 'negative' polarities of sentiment.…”

Section: ) Datasetmentioning

confidence: 99%

HMTL: Heterogeneous Modality Transfer Learning for Audio-Visual Sentiment Analysis

Seo

Kim

2020

IEEE Access

View full text Add to dashboard Cite

Multimodal sentiment analysis is an extended approach to traditional language-based sentiment analysis, which uses other relevant modality data. Multimodal sentiment analysis usually applies visual, textual, and acoustic representations for sentiment prediction. Recently, various data fusion methodologies have been proposed for multimodal sentiment analysis. In most cases, textual modality plays a major role, and visual and acoustic modalities are used as auxiliary sources for multimodal sentiment analysis. However, in general multimedia such as video, text transcripts of an individual's speech are not provided. Research on an audiovisual sentiment analysis methodology that does not depend on text modality is essential for multimodal sentiment analysis in real-world industrial applications. Therefore, it is important to improve audiovisual sentiment analysis because it currently exhibits lower performance than multimodal sentiment analysis, including text modality. In this paper, we propose heterogeneous modality transfer learning (HMTL) to utilize the knowledge of aligned text data as a source modality in transfer learning to improve audiovisual sentiment analysis performance. Our approach uses a decoder and adversarial learning techniques to reduce the gap between the source and target modalities in the embedded space for multimodal representation. Our proposed methodology experimentally outperformed recent unimodal and bimodal audiovisual sentiment analysis achievements.

show abstract

Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis

Cited by 146 publications

References 8 publications

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos

HMTL: Heterogeneous Modality Transfer Learning for Audio-Visual Sentiment Analysis

Contact Info

Product

Resources

About