Analyzing Multimodal Sentiment Via Acoustic- and Visual-LSTM With Channel-Aware Temporal Convolution Network

Mai, Sijie; Xing, Songlong; Hu, Haifeng

doi:10.1109/taslp.2021.3068598

Cited by 66 publications

(20 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We can infer from Fig. 2 that, the language modality is the most informative modality that is rarely filtered out (and this conclusion is consistent with other works (Mai et al, 2021b)). Contrary to it, the acoustic modality is frequently identified as noisy and filtered out which is the most uninformative modality.…”

Section: Analysis On the Modality Importancesupporting

confidence: 90%

“…Moreover, even with satisfactory unimodal networks, it is not always the case that multimodal models reach higher performance than the unimodal ones (Mai et al, 2021b). The reason may be that, a modality may not contain useful information in some utterances and may even carry noises, which hinders the learning of correct multimodal embedding.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis

Zeng¹,

Mai²,

Hu³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Section: Analysis On the Modality Importancesupporting

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis

Zeng¹,

Mai²,

Hu³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

“…CMU-MOSEI includes labels not only for the emotion recognition task but also for the (text-based) sentiment analysis problem. For this reason, it is mostly used for sentiment analysis architectures (e.g., [63,66,67,70,[83][84][85]). However, its benefits must be adopted by researchers on emotion recognition in order to produce more robust models and gain a better insight into their actual performance.…”

Section: Datasetsmentioning

confidence: 99%

Deep Multimodal Emotion Recognition on Human Speech: A Review

Koromilas

Γιαννακόπουλος

2021

Applied Sciences

View full text Add to dashboard Cite

This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness.

show abstract

“…However, this study, due to different pretraining, reports smaller amount of parameters for some models (eg. [23] reports 1,549,321 parameters for MulT on the MO-SEI dataset with a different pretraining procedure). However, for the sake of a complete comparison, we choose to report the results of [21], though this may underestimate the number of parameters of the reported models.…”

Section: Model Complexitymentioning

confidence: 99%

Unsupervised Multimodal Language Representations using Convolutional Autoencoders

Koromilas¹,

Giannakopoulos²

2021

Preprint

View full text Add to dashboard Cite

Multimodal Language Analysis is a demanding area of research, since it is associated with two requirements: combining different modalities and capturing temporal information. During the last years, several works have been proposed in the area, mostly centered around supervised learning in downstream tasks. In this paper we propose extracting unsupervised Multimodal Language representations that are universal and can be applied to different tasks. Towards this end, we map the word-level aligned multimodal sequences to 2-D matrices and then use Convolutional Autoencoders to learn embeddings by combining multiple datasets. Extensive experimentation on Sentiment Analysis (MOSEI) and Emotion Recognition (IEMOCAP) indicate that the learned representations can achieve near-state-of-the-art performance with just the use of a Logistic Regression algorithm for downstream classification. It is also shown that our method is extremely lightweight and can be easily generalized to other tasks and unseen data with small performance drop and almost the same number of parameters. The proposed multimodal representation models are open-sourced and will help grow the applicability of Multimodal Language.

show abstract

Analyzing Multimodal Sentiment Via Acoustic- and Visual-LSTM With Channel-Aware Temporal Convolution Network

Cited by 66 publications

References 40 publications

Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis

Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis

Deep Multimodal Emotion Recognition on Human Speech: A Review

Unsupervised Multimodal Language Representations using Convolutional Autoencoders

Contact Info

Product

Resources

About