Tensor Fusion Network for Multimodal Sentiment Analysis

Zadeh, Amir; Chen, Minghai; Poria, Soujanya; Wang, Zhaoxia; Morency, Louis–Philippe

doi:10.18653/v1/d17-1115

Cited by 1,056 publications

(487 citation statements)

References 51 publications

Supporting

Mentioning

482

Contrasting

Unclassified

Order By: Relevance

“…Recently, Poria et al successfully used RNN‐based deep networks for multimodal emotion recognition and it was followed by other works by Zadeh et al; Hazarika et al, used memory networks for emotion recognition in dyadic conversations, where two distinct memory networks enabled interspeaker interaction, yielding state‐of‐the‐art performance.…”

Section: Related Workmentioning

confidence: 99%

Bimodal recognition of affective states with the features inspired from human visual and auditory perception system

Arivazhagan¹,

Russel²

2019

Int J Imaging Syst Tech

View full text Add to dashboard Cite

In this article, attention‐based mechanism with the enhancement on biologically inspired network for emotion recognition is proposed. Existing bio‐inspired models use multiscale and multiorientation architecture to gain discriminative power and to extract meticulous visual features. Prevailing HMAX model represents S2 layers by randomly selected prototype patches from training samples that increase the computational complexity and degrade the discerning ability. As eyes and mouth regions are the most powerful and reliable cues in determining facial emotions, they serve as the prototype patches for S2 layer in HMAX model. Audio code 4 book is constructed from mel‐frequency cepstral coefficients, temporal and spectral features processed by principal component analysis. Audio and video data features are fused to train support vector machine classifier. The attained results on eNTERFACE, surrey audio‐visual expressed emotion and acted facial expressions in the wild database datasets ascertain the efficiency of the proposed architecture for emotion recognition.

show abstract

Section: Related Workmentioning

confidence: 99%

Bimodal recognition of affective states with the features inspired from human visual and auditory perception system

Arivazhagan¹,

Russel²

2019

Int J Imaging Syst Tech

View full text Add to dashboard Cite

show abstract

“…There are many commercial and social applications related to sentiment analysis. Actually, sentiment analysis can utilize multimodal data including text, speech and video [33]- [35]. In natural language processing, a basic sentiment analysis task is to classify the polarity of a given text at the document, sentence, or feature and aspect level.…”

Section: B Sentiment Analysis (Sa)mentioning

confidence: 99%

A Sequential Neural Encoder With Latent Structured Description for Modeling Sentences

Ruan

Qian

Ling

2018

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Abstract-In this paper, we propose a sequential neural encoder with latent structured description (SNELSD) for modeling sentences. This model introduces latent chunk-level representations into conventional sequential neural encoders, i.e., recurrent neural networks (RNNs) with long short-term memory (LSTM) units, to consider the compositionality of languages in semantic modeling. An SNELSD model has a hierarchical structure that includes a detection layer and a description layer. The detection layer predicts the boundaries of latent word chunks in an input sentence and derives a chunk-level vector for each word. The description layer utilizes modified LSTM units to process these chunk-level vectors in a recurrent manner and produces sequential encoding outputs. These output vectors are further concatenated with word vectors or the outputs of a chain LSTM encoder to obtain the final sentence representation. All the model parameters are learned in an end-to-end manner without a dependency on additional text chunking or syntax parsing. A natural language inference (NLI) task and a sentiment analysis (SA) task are adopted to evaluate the performance of our proposed model. The experimental results demonstrate the effectiveness of the proposed SNELSD model on exploring taskdependent chunking patterns during the semantic modeling of sentences. Furthermore, the proposed method achieves better performance than conventional chain LSTMs and tree-structured LSTMs on both tasks.Index Terms-recurrent neural network, long short-term memory, sentence modeling, syntax structure.

show abstract

“…Methods that learn the modalities independently and fuse the output of modality specific representations [1,2], 2. Methods that jointly learn the interactions between two or three modalities [3,4], and 3. Methods that explicitly learn contributions from these unimodal and cross modal cues, typically using attention based techniques [5,6,7,8,9,10].…”

Section: Introductionmentioning

confidence: 99%

“…Most of the existing approaches propose either fusion at different granularities [3,9] or use a cross interaction block that couple the features from different modalities [10,6].…”

Section: Introductionmentioning

confidence: 99%

Gated Mechanism for Attention Based Multi Modal Sentiment Analysis

Kumar¹,

Vepa²

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Multimodal sentiment analysis has recently gained popularity because of its relevance to social media posts, customer service calls and video blogs. In this paper, we address three aspects of multimodal sentiment analysis; 1. Cross modal interaction learning, i.e. how multiple modalities contribute to the sentiment, 2. Learning long-term dependencies in multimodal interactions and 3. Fusion of unimodal and cross modal cues. Out of these three, we find that learning cross modal interactions is beneficial for this problem. We perform experiments on two benchmark datasets, CMU Multimodal Opinion level Sentiment Intensity (CMU-MOSI) and CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) corpus. Our approach on both these tasks yields accuracies of 83.9% and 81.1% respectively, which is 1.6% and 1.34% absolute improvement over current state-ofthe-art.

show abstract

Tensor Fusion Network for Multimodal Sentiment Analysis

Cited by 1,056 publications

References 51 publications

Bimodal recognition of affective states with the features inspired from human visual and auditory perception system

Bimodal recognition of affective states with the features inspired from human visual and auditory perception system

A Sequential Neural Encoder With Latent Structured Description for Modeling Sentences

Gated Mechanism for Attention Based Multi Modal Sentiment Analysis

Contact Info

Product

Resources

About