Multi-Modal Sentiment Analysis Using Deep Canonical Correlation Analysis

Sun, Zhan-Li; Sarma, Prathusha Kameswara; Sethares, William A.; Bucy, Erik P.

doi:10.21437/interspeech.2019-2482

Cited by 21 publications

(18 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use the CCA to learn a new common space for audio and video modes, and combine the learned audio and video features with the original text embedding. This is because, (Sun et al 2019) showed that using a CCA-based method to correlate audio and video is more effective that correlating audiotext or video-text. • Kernel-CCA: Kernel-CCA (Akaho 2006) introduces a nonlinearity via kernel maps.…”

Section: Baseline Methodsmentioning

confidence: 99%

“…• DCCA: A Deep CCA based algorithm proposed by (Sun et al 2019). Audio and video features are simply concatenated and then be correlated with text features using DCCA.…”

Section: Baseline Methodsmentioning

confidence: 99%

“…Models proposed by (Rotman, Vulić, and Reichart 2018) use Deep Partial Canonical Correlation Analysis (DPCCA), a variant of DCCA, for studying the relationship between two languages based on the same image they are describing. Work by (Sun et al 2019) investigates the application of DCCA to simple concatenations of multimodal-features, while (Hazarika et al 2018) applied CCA methods to learn joint-representation for detecting sarcasm. Both approaches show the effectiveness of CCA methods towards learning potential correlation between two input variables.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis

Sun

Sarma²,

Sethares

et al. 2020

AAAI

Self Cite

249

View full text Add to dashboard Cite

Multimodal language analysis often considers relationships between features based on text and those based on acoustical and visual properties. Text features typically outperform non-text features in sentiment analysis or emotion recognition tasks in part because the text features are derived from advanced language models or word embeddings trained on massive data sources while audio and video features are human-engineered and comparatively underdeveloped. Given that the text, audio, and video are describing the same utterance in different ways, we hypothesize that the multimodal sentiment analysis and emotion recognition can be improved by learning (hidden) correlations between features extracted from the outer product of text and audio (we call this text-based audio) and analogous text-based video. This paper proposes a novel model, the Interaction Canonical Correlation Network (ICCN), to learn such multimodal embeddings. ICCN learns correlations between all three modes via deep canonical correlation analysis (DCCA) and the proposed embeddings are then tested on several benchmark datasets and against other state-of-the-art multimodal embedding algorithms. Empirical results and ablation studies confirm the effectiveness of ICCN in capturing useful information from all three views.

show abstract

Section: Baseline Methodsmentioning

confidence: 99%

“…• DCCA: A Deep CCA based algorithm proposed by (Sun et al 2019). Audio and video features are simply concatenated and then be correlated with text features using DCCA.…”

Section: Baseline Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis

Sun

Sarma²,

Sethares

et al. 2020

AAAI

Self Cite

249

View full text Add to dashboard Cite

show abstract

“…Following [13][14], we use Canonical Correlation Analysis (CCA) for feature alignment, such that L al = -CCA. CCA for deep neural networks, also known as Deep CCA or DCCA, is a method to learn complex nonlinear transformations of data from two different modalities, such that the resulting representations are highly linearly correlated [15].…”

Section: Proposed Methodsmentioning

confidence: 99%

Robust Latent Representations Via Cross-Modal Translation and Alignment

Rajan

Brutti

Cavallaro

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Multi-modal learning relates information across observation modalities of the same physical phenomenon to leverage complementary information. Most multi-modal machine learning methods require that all the modalities used for training are also available for testing. This is a limitation when signals from some modalities are unavailable or severely degraded. To address this limitation, we aim to improve the testing performance of uni-modal systems using multiple modalities during training only. The proposed multi-modal training framework uses cross-modal translation and correlation-based latent space alignment to improve the representations of a worse performing (or weaker) modality. The translation from the weaker to the better performing (or stronger) modality generates a multi-modal intermediate encoding that is representative of both modalities. This encoding is then correlated with the stronger modality representation in a shared latent space. We validate the proposed framework on the AVEC 2016 dataset (RECOLA) for continuous emotion recognition and show the effectiveness of the framework that achieves state-ofthe-art (uni-modal) performance for weaker modalities.

show abstract

“…[5] 76.5 73. 4 Zadeh et al [7] 76.9 77.0 Georgiou et al [9] 76.9 76.9 Poria et al [2] 77.64 -Ghosal et al [10] 82.31 80.69 Ghosal et al [10] 79.80 -Sun et al [4] 80 [7], ( ¦ ) results are obtained on CMU-MOSEI dataset after excluding the utterances with sentiment score of 0. We mention the results of proposed model with this setup in the parenthesis.…”

Section: Cmu-mosei Approachmentioning

confidence: 99%

Gated Mechanism for Attention Based Multi Modal Sentiment Analysis

Kumar¹,

Vepa²

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Multimodal sentiment analysis has recently gained popularity because of its relevance to social media posts, customer service calls and video blogs. In this paper, we address three aspects of multimodal sentiment analysis; 1. Cross modal interaction learning, i.e. how multiple modalities contribute to the sentiment, 2. Learning long-term dependencies in multimodal interactions and 3. Fusion of unimodal and cross modal cues. Out of these three, we find that learning cross modal interactions is beneficial for this problem. We perform experiments on two benchmark datasets, CMU Multimodal Opinion level Sentiment Intensity (CMU-MOSI) and CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) corpus. Our approach on both these tasks yields accuracies of 83.9% and 81.1% respectively, which is 1.6% and 1.34% absolute improvement over current state-ofthe-art.

show abstract

Multi-Modal Sentiment Analysis Using Deep Canonical Correlation Analysis

Cited by 21 publications

References 16 publications

Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis

Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis

Robust Latent Representations Via Cross-Modal Translation and Alignment

Gated Mechanism for Attention Based Multi Modal Sentiment Analysis

Contact Info

Product

Resources

About