Improving the Modality Representation with multi-view Contrastive Learning for Multimodal Sentiment Analysis

Li, Peipei; Zheng, Xin; Li, Hong; Liu, Jie; Ren, Yimo; Zhu, Hongsong; Sun, Lijun

doi:10.1109/icassp49357.2023.10096470

Cited by 3 publications

(1 citation statement)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…( 6) SK-GCN [24]: This model combines syntactic dependency trees and knowledge graphs to effectively integrate syntactic knowledge with external knowledge. ( 7) MVCL [34]: This is a novel framework with multi-view contrastive learning for improving the modality representation used for the multimodal sentiment analysis. ( 8) AMLT [35]: A novel Adaptive Language-guided Multimodal Transformer (ALMT) is proposed to better model sentiment cues for robust Multimodal Sentiment Analysis (MSA).…”

Section: Baselinesmentioning

confidence: 99%

Multichannel Multimodal Emotion Analysis of Cross-Modal Feedback Interactions Based on Knowledge Graph

Dong,

Fan,

2024

Neural Process Lett

View full text Add to dashboard Cite

Multimodal sentiment analysis is a downstream branch task of sentiment analysis with high attention at present. Previous work in multimodal sentiment analysis have focused on the representation and fusion of modalities, capturing the underlying semantic relationships between modalities by considering contextual information. While this approach is feasible for simple contextual comments, more complex comments require the integration of external knowledge to obtain more accurate sentiment information. However, incorporating external knowledge into sentiment analysis to enhance information complementarity has not been thoroughly investigated. To address this, we propose a multichannel cross-modal feedback interaction model that incorporates the knowledge graph into multimodal sentiment analysis. Our proposed model consists of two main components: the cross-modal feedback recurrent interaction module and the external knowledge module for capturing latent information. The cross-modal interaction employs a self-feedback mechanism during network training, extracting feature representations of each modality and using these representations to mask sensory inputs, allowing the model to perform feedback-based feature masking. The external knowledge graph captures potential semantic information representations in the textual data through knowledge graph embedding. Finally, a global feature fusion module is employed for multichannel multimodal information integration. On two publicly available datasets, our method demonstrates good performance in terms of accuracy and F1 scores, compared to state-of-the-art models and several baselines.

show abstract

Section: Baselinesmentioning

confidence: 99%