MSA-HCL: Multimodal sentiment analysis model with hybrid contrastive learning

Abstract

Multimodal sentiment analysis (MSA) has become a popular research topic due to the rapid growth of user-generated data on the Internet. However, it remains challenging to capture the interaction between modalities. To address this issue, a Multimodal Sentiment Analysis model with Hybrid Contrastive Learning (MSA-HCL) is proposed in this study by leveraging a hybrid contrastive learning model. The proposed MSA-HCL model is able to fuse different modalities based on their contextual similarity by incorporating inter-modal contrastive learning and intra-modal contrastive learning in a hybrid manner. The MSA-HCL model consists of three submodules. The inter-modal contrastive learning module performs the alignment of bimodal inputs with similar semantics and learns their interdependence. The intra-modal contrastive learning module preserves the modality-specific representations by learning similarity within the same modality. In order to reduce the noise introduced by multimodal fusion, a supervised contrastive learning module is employed to learn the consistency between fused modality and textual modality. To evaluate the performance of the proposed MSA-HCL model, extensive comparison experiments and ablation studies are conducted on two benchmark datasets, and the experimental results show that MSA-HCL achieves state-ofthe-art performance. Some useful insights are derived from the ablation results as well.