Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data. This research area's major concern lies in developing an extraordinary fusion scheme that can extract and integrate key information from various modalities. However, previous work is restricted by the lack of leveraging dynamics of independence and correlation between modalities to reach top performance. To mitigate this, we propose the Bi-Bimodal Fusion Network (BBFN), a novel endto-end network that performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. The two parts are trained simultaneously such that the combat between them is simulated. The model takes two bimodal pairs as input due to the known information imbalance among modalities. In addition, we leverage a gated control mechanism in the Transformer architecture to further improve the final output. Experimental results on three datasets (CMU-MOSI, CMU-MOSEI, and UR-FUNNY) verifies that our model significantly outperforms the SOTA.

Modality-Invariant and -Specific Representations with Crossmodal Transformer for Multimodal Sentiment Analysis

“…• LMF: The model uses tensors to explore the interactions between modes and uses low-rank decomposition to alleviate the problem of number of parameters. • MFM: To enhance the robustness of the model of capturing intra-and inter modality dynamics, MFM is a cycle style generative-discriminative model [14]. • MulT: Multimodal Transformer constructs an architecture unimodal and crossmodal transformer networks and complete fusion process by attention [15].…”

“…The CMU-MOSEI dataset is an upgraded version of CMU-MOSI concerning the number of samples. It is also enriched in terms of the versatility of speakers and covers a broader scope of topics [14].…”

“…Deep learning based multimodal fusion algorithms can be classified according to the level of fusion: pixel level, feature level and decision level. Most of the advanced fusion methods are currently based on feature level, such as the MISA model for modality-invariant and -specific representations [13], the BBFN model for bi-bimodal modality fusion for correlation-controlled [14], and the MMIM model for improving multimodal fusion with hierarchical mutual information maximization [15]. The MISA-CT model proposed in this paper is based on the MISA model, which follows the basic process of MISA for multimodal data, mapping the original modal data to modality-invariant and -specific subspaces respectively.…”

“…It can be intuitively seen that the replication results of each paper for the MISA model are not exactly the same, and different hardware devices and software versions make the experimental results of the model not fully replicated, but the experimental results differ very little. In order to increase the credibility of the experiments, a set of experimental results of MISA in paper[14] and the best MISA-CT model in this paper are added to compare the experimental results, as shown in Table2. Replication results of different papers for the MISA model.…”

LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition

“…Most of the high performance of existing models are dependent on a great number of learnable parameters [ 15 , 16 ], ignoring the potential application in some promising areas like human–computer interaction (HCI), which requires real-time and light models. Thus, a lightweight model is necessary to improve the feasibility and practicability of the application of speech emotion recognition.…”