Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities

Zeng, Jiandian; Liu, Tianyi; Zhou, Jiantao

doi:10.1145/3477495.3532064

Cited by 32 publications

(4 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To this end, recent works [22,23,39,40] explore to build multimodal models which are robust to data with missing modalities. SMIL [23] is proposed to estimate the latent features of the modality-incomplete data via Bayesian Meta-Learning.…”

Section: Related Workmentioning

confidence: 99%

“…SMIL [23] is proposed to estimate the latent features of the modality-incomplete data via Bayesian Meta-Learning. Zeng et al [39] propose a tag encoding module to assist the transformer's encoder learning with different missing modalities. MMIN [40] predicts the representation of any missing modality given other available modalities via learning a joint multimodal representations.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Prompting with Missing Modalities for Visual Recognition

Lee¹,

Tsai²,

Chiu³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models. To this end, we propose to utilize prompt learning and mitigate the above two challenges together. Specifically, our modality-missing-aware prompts can be plugged into multimodal transformers to handle general missing-modality cases, while only requiring less than 1% learnable parameters compared to training the entire model. We further explore the effect of different prompt configurations and analyze the robustness to missing modality. Extensive experiments are conducted to show the effectiveness of our prompt learning framework that improves the performance under various missing-modality cases, while alleviating the requirement of heavy model re-training. Code is available. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Multimodal Prompting with Missing Modalities for Visual Recognition

Lee¹,

Tsai²,

Chiu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…MMIM (Han et al, 2021) preserves taskrelated information through hierarchical maximization of Mutual Information (MI) between pairs of unimodal inputs and the resulting fusion of multiple modalities. Recently, several works (Ma et al, 2021a;Sun et al, 2022;Zeng et al, 2022) have focused on uncertainly solving the missing modalities problem.…”

Section: Related Workmentioning

confidence: 99%

Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation

Nguyen,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

The effectiveness of a model is heavily reliant on the quality of the fusion representation of multiple modalities in multimodal sentiment analysis. Moreover, each modality is extracted from raw input and integrated with the rest to construct a multimodal representation. Although previous methods have proposed multimodal representations and achieved promising results, most of them focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. Additionally, they fail to capture the significance of unimodal representations in the fusion vector. To address these limitations, we introduce a framework called Supervised Angularbased Contrastive Learning for Multimodal Sentiment Analysis. This framework aims to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector's modality. Our experimental results, along with visualizations on two widely used datasets, demonstrate the effectiveness of our approach.

show abstract

“…This technique further enhances the model's capacity to make accurate emotional distinctions. As such, MER has captured the attention of many researchers, aiming to combine information from various modalities, which can corroborate and complement each other, thus providing more comprehensive and accurate information for emotional judgment and significantly improving the performance of emotional judgments [12][13][14][15].…”

Section: Introductionmentioning

confidence: 99%

A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face

Lian,

Lu,

et al. 2023

Entropy

View full text Add to dashboard Cite

Multimodal emotion recognition (MER) refers to the identification and understanding of human emotional states by combining different signals, including—but not limited to—text, speech, and face cues. MER plays a crucial role in the human–computer interaction (HCI) domain. With the recent progression of deep learning technologies and the increasing availability of multimodal datasets, the MER domain has witnessed considerable development, resulting in numerous significant research breakthroughs. However, a conspicuous absence of thorough and focused reviews on these deep learning-based MER achievements is observed. This survey aims to bridge this gap by providing a comprehensive overview of the recent advancements in MER based on deep learning. For an orderly exposition, this paper first outlines a meticulous analysis of the current multimodal datasets, emphasizing their advantages and constraints. Subsequently, we thoroughly scrutinize diverse methods for multimodal emotional feature extraction, highlighting the merits and demerits of each method. Moreover, we perform an exhaustive analysis of various MER algorithms, with particular focus on the model-agnostic fusion methods (including early fusion, late fusion, and hybrid fusion) and fusion based on intermediate layers of deep models (encompassing simple concatenation fusion, utterance-level interaction fusion, and fine-grained interaction fusion). We assess the strengths and weaknesses of these fusion strategies, providing guidance to researchers to help them select the most suitable techniques for their studies. In summary, this survey aims to provide a thorough and insightful review of the field of deep learning-based MER. It is intended as a valuable guide to aid researchers in furthering the evolution of this dynamic and impactful field.

show abstract

Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities

Cited by 32 publications

References 47 publications

Multimodal Prompting with Missing Modalities for Visual Recognition

Multimodal Prompting with Missing Modalities for Visual Recognition

Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation

A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face

Contact Info

Product

Resources

About