Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis

Yu, Wenmeng; Xu, Hua; Yuan, Zhiquan; Wu, Jiele

doi:10.48550/arxiv.2102.04830

Cited by 10 publications

(21 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dai et al (2021a) proposed a multi-task learning approach using weak supervision for multimodal emotion recognition. Yu et al (2021) proposed a way to fuse features from different modalities by combining self-supervised and multi-task learning. Although selfsupervised and multi-task learning can effectively alleviate the problem of small samples, how to perform efficient cross-modal interactions is still a tremendously challenging issue for researchers.…”

Section: Related Workmentioning

confidence: 99%

“…In the previous works (Sahay et al 2020;Rahman et al 2020;Hazarika, Zimmermann, and Poria 2020;Yu et al 2021;Dai et al 2021a), Transformers (Vaswani et al 2017) are mostly used for unaligned multimodal emotion recognition. Typically, Tsai et al (2019a) proposed the Multimodal Transformer (MulT) method to fuse information from different modalities in unaligned sequences without explicitly aligning the data.…”

Section: Introductionmentioning

confidence: 99%

“…Comparison on the CMU-MOSEI dataset under both word-aligned setting and unaligned setting. *: reproduced from open-source code; †: from(Lv et al 2021); ‡: from(Yu et al 2021). LMR-CBT achieves comparable and superior performance with only 0.41M parameters.…”

mentioning

confidence: 99%

See 2 more Smart Citations

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Fu¹,

Liu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message hub to fuse language, visual, and audio modalities. However, those approaches introduce information redundancy when fusing features and are inefficient without considering the complementarity of modalities. In this paper, we propose an efficient neural network to learn modality-fused representations with CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned multimodal sequences. Specifically, we first perform feature extraction for the three modalities respectively to obtain the local structure of the sequences. Then, we design a novel transformer with cross-modal blocks (CB-Transformer) that enables complementary learning of different modalities, mainly divided into local temporal learning, cross-modal feature fusion and global self-attention representations. In addition, we splice the fused features with the original features to classify the emotions of the sequences. Finally, we conduct word-aligned and unaligned experiments on three challenging datasets, IEMOCAP, CMU-MOSI, and CMU-MOSEI. The experimental results show the superiority and efficiency of our proposed method in both settings. Compared with the mainstream methods, our approach reaches the state-of-the-art with a minimum number of parameters.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Fu¹,

Liu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Existing methods to learn unified representations are grouped in two categories: through loss back-propagation or geometric manipulation in the feature spaces. The former only tunes the parameters based on back-propagated gradients from the task loss Tsai et al, 2019a;, reconstruction loss (Mai et al, 2020), or auxiliary task loss Yu et al, 2021). The latter additionally rectifies the spatial orientation of unimodal or multimodal representations by matrix decomposition (Liu et al, 2018) or Euclidean measure optimization (Sun et al, 2020;.…”

Section: Introductionmentioning

confidence: 99%

Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

Wang¹,

Chen²,

Poria³

2021

Preprint

View full text Add to dashboard Cite

In multimodal sentiment analysis (MSA), the performance of a model highly depends on the quality of synthesized embeddings. These embeddings are generated from the upstream process called multimodal fusion, which aims to extract and combine the input unimodal raw data to produce a richer multimodal representation. Previous work either back-propagates the task loss or manipulates the geometric property of feature spaces to produce favorable fusion results, which neglects the preservation of critical task-related information that flows from input to the fusion results. In this work, we propose a framework named MultiModal InfoMax (MMIM), which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain taskrelated information through multimodal fusion. The framework is jointly trained with the main task (MSA) to improve the performance of the downstream MSA task. To address the intractable issue of MI bounds, we further formulate a set of computationally simple parametric and non-parametric methods to approximate their truth value. Experimental results on the two widely used datasets demonstrate the efficacy of our approach.

show abstract

“…Previous work (Liu and Zhang, 2012;Tian et al, 2020) mainly focused on text sentiment analysis and achieved promising results. Recently, with the development of short video applications, multimodal sentiment analysis has obtained more attention (Tsai et al, 2019;Yu et al, 2021) and a lots of datasets (Li et al, 2017;Poria et al, 2018; are proposed to advance its developments. However, current multimodal sentiment analysis datasets usually follow the traditional sentiment system (positive, neutral and negative ) or emotion system (happy, sad, surprise and so on), which is far from satisfactory especially for video recommendation scenario of application.…”

Section: Introductionmentioning

confidence: 99%

A Multimodal Sentiment Dataset for Video Recommendation

Tang¹,

Líu²,

Xiao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, multimodal sentiment analysis has seen remarkable advance and a lot of datasets are proposed for its development. In general, current multimodal sentiment analysis datasets usually follow the traditional system of sentiment/emotion, such as positive, negative and so on. However, when applied in the scenario of video recommendation, the traditional sentiment/emotion system is hard to be leveraged to represent different contents of videos in the perspective of visual senses and language understanding. Based on this, we propose a multimodal sentiment analysis dataset, named baiDu Video Sentiment dataset (Du-VideoSenti), and introduce a new sentiment system which is designed to describe the sentimental style of a video on recommendation scenery. Specifically, DuVideoSenti consists of 5,630 videos which displayed on Baidu 1 , each video is manually annotated with a sentimental style label which describes the user's real feeling of a video. Furthermore, we propose UNIMO (Li et al., 2020) as our baseline for DuVideoSenti. Experimental results show that DuVideoSenti brings new challenges to multimodal sentiment analysis, and could be used as a new benchmark for evaluating apporaches designed for video understanding and multimodal fusion. We also expect our proposed DuVideoSenti could further improve the development of multimodal sentiment analysis and its application to video recommendations.

show abstract

Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis

Cited by 10 publications

References 15 publications

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

A Multimodal Sentiment Dataset for Video Recommendation

Contact Info

Product

Resources

About