Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis

Abstract

Representation Learning is a significant and challenging task in multimodal learning. Effective modality representations should contain two parts of characteristics: the consistency and the difference. Due to the unified multimodal annota- tion, existing methods are restricted in capturing differenti- ated information. However, additional unimodal annotations are high time- and labor-cost. In this paper, we design a la- bel generation module based on the self-supervised learning strategy to acquire independent unimodal supervisions. Then, joint training the multimodal and uni-modal tasks to learn the consistency and difference, respectively. Moreover, dur- ing the training stage, we design a weight-adjustment strat- egy to balance the learning progress among different sub- tasks. That is to guide the subtasks to focus on samples with the larger difference between modality supervisions. Last, we conduct extensive experiments on three public multimodal baseline datasets. The experimental results validate the re- liability and stability of auto-generated unimodal supervi- sions. On MOSI and MOSEI datasets, our method surpasses the current state-of-the-art methods. On the SIMS dataset, our method achieves comparable performance than human- annotated unimodal labels. The full codes are available at https://github.com/thuiar/Self-MM.

CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations

“…Early fusion methods that concatenate multimodal data at the input level, such as [10], [12], can learn inter-modality dynamics, but they have limitations in learning intra-modality dynamics. Late fusion methods that integrate different modalities at the prediction level, such as [13], [14], focus on modeling intra-modality dynamics rather than inter-modality dynamics. Some studies introduced additional modules to model both intra-modality and inter-modality dynamics.…”

“…MISA [7] introduced a modality-invariant encoder that projects each modality to a common space and used Central Moment Discrepancy (CMD) [8] as a similarity loss to align cross-modal representations within the common space. Self-MM [14] presented a label generation module based on self-supervised learning to acquire independent unimodal supervisions. There were also studies focused on changes in word representations.…”

“…Linguistic Embedding Subnetwork: Following recent approaches [2], [7], [14], we utilize the pre-trained BERT's embedding as the feature extractor for word tokens. Given a length N word tokens W = [W 1 , W 2 , ..., W N ] ∈ R N , we input W to the input embedder which outputs X = [X 1 , X 2 , ..., X N ] ∈ R N ×d l .…”

“…MAG-BERT (Multimodal Adaptation Gate for Bert) [2] attached Multimodal Adaptation Gate (MAG) to BERT to accept multimodal non-verbal information during fine-tuning. Self-MM (Self-Supervised Multi-task Multimodal sentiment analysis network) [14] designed a label generation module based on the self-supervised learning strategy to acquire independent unimodal supervisions.…”

“…Evaluation Metrics. Following the previous works [7], [14], we perform two different evaluation tasks on CMU-MOSI and CMU-MOSEI: classification and regression. We report the weighted F1 score (F1-Score) and binary classification accuracy (Acc-2) for classification.…”