TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis

Wang, Di; Guo, Xutong; Tian, Yumin; Liu, Jinhui; He, LiHuo; Luo, Xiangfeng

doi:10.1016/j.patcog.2022.109259

Cited by 80 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In text-image person re-identification, Yan et al 29 launched a fine-grained information excavation framework which is driven by contrastive language-image pretraining. As for multimodal sentiment analysis, Wang et al 30 proposed the text enhanced transformer fusion network which can achieve effective unified multimodal representations.…”

Section: Multi-modal Joint Decisionmentioning

confidence: 99%

CDN: Collaborate Decision Network Based on Cross-modal Attention for Social Media Microblog Recognition

Peng,

Li,

Fang

2024

Preprint

View full text Add to dashboard Cite

Recently, social media has gradually become an important topic in public opinion analysis field. Among social media, Microblog is one of the most important platforms because it is short, convenient, mobile and instantaneous. Social media microblog recognition well reflects the attitudes from an enormous big colony to a specific incident, either positive or negative, which can be used for deriving competitive intelligence, marketing strategies, detecting depression and so on. However, the existing methods usually use only text or image from internet but not take advantages of their complementary information to finalize the recognition, it limits the performance and robustness of the algorithms. In this paper, we present a collaborate decision network (CDN) based on cross-modal attention to exploit the discriminative attributes of multi modalities by data- and knowledge joint driven strategy in depth, and further improve the recognition performance. In addition, we collect and construct a visual-text microblog recognition dataset with 2854 samples to support the subsequent research of related fields. Finally, experimental reuslts on the collected dataset show the effectiveness and superiority of the proposed CDN.

show abstract

Section: Multi-modal Joint Decisionmentioning

confidence: 99%

CDN: Collaborate Decision Network Based on Cross-modal Attention for Social Media Microblog Recognition

Peng,

Li,

Fang

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…More recently, Wang et al [52] proposed a novel Text Enhanced Transformer Fusion Network (TETFN) method that learns text-oriented pairwise cross-modal mappings to obtain effective unified multi-modal representations. Yang et al [53] applied BERT to translate visual and audio features into text features to enhance the quality of both visual and audio features.…”

Section: Multi-modal Sentiment Analysismentioning

confidence: 99%

Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis

He,

Zhang,

Song

et al. 2024

BDCC

View full text Add to dashboard Cite

With the popularization of better network access and the penetration of personal smartphones in today’s world, the explosion of multi-modal data, particularly opinionated video messages, has created urgent demands and immense opportunities for Multi-Modal Sentiment Analysis (MSA). Deep learning with the attention mechanism has served as the foundation technique for most state-of-the-art MSA models due to its ability to learn complex inter- and intra-relationships among different modalities embedded in video messages, both temporally and spatially. However, modal fusion is still a major challenge due to the vast feature space created by the interactions among different data modalities. To address the modal fusion challenge, we propose an MSA algorithm based on deep learning and the attention mechanism, namely the Mixture of Attention Variants for Modal Fusion (MAVMF). The MAVMF algorithm includes a two-stage process: in stage one, self-attention is applied to effectively extract image and text features, and the dependency relationships in the context of video discourse are captured by a bidirectional gated recurrent neural module; in stage two, four multi-modal attention variants are leveraged to learn the emotional contributions of important features from different modalities. Our proposed approach is end-to-end and has been shown to achieve a superior performance to the state-of-the-art algorithms when tested with two largest public datasets, CMU-MOSI and CMU-MOSEI.

show abstract

“…TETFN [26] provided a text-enhanced transformer fusion network that captured the relevance between text and image through bidirectional attention mechanisms and enhanced the semantic representations of images using textual features.…”

Section: Evaluation Metricsmentioning

confidence: 99%

MSA-HCL: Multimodal sentiment analysis model with hybrid contrastive learning

Zhao,

Zhang,

Hua

et al. 2024

MFC

View full text Add to dashboard Cite

Multimodal sentiment analysis (MSA) has become a popular research topic due to the rapid growth of user-generated data on the Internet. However, it remains challenging to capture the interaction between modalities. To address this issue, a Multimodal Sentiment Analysis model with Hybrid Contrastive Learning (MSA-HCL) is proposed in this study by leveraging a hybrid contrastive learning model. The proposed MSA-HCL model is able to fuse different modalities based on their contextual similarity by incorporating inter-modal contrastive learning and intra-modal contrastive learning in a hybrid manner. The MSA-HCL model consists of three submodules. The inter-modal contrastive learning module performs the alignment of bimodal inputs with similar semantics and learns their interdependence. The intra-modal contrastive learning module preserves the modality-specific representations by learning similarity within the same modality. In order to reduce the noise introduced by multimodal fusion, a supervised contrastive learning module is employed to learn the consistency between fused modality and textual modality. To evaluate the performance of the proposed MSA-HCL model, extensive comparison experiments and ablation studies are conducted on two benchmark datasets, and the experimental results show that MSA-HCL achieves state-ofthe-art performance. Some useful insights are derived from the ablation results as well.

show abstract

TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis

Cited by 80 publications

References 11 publications

CDN: Collaborate Decision Network Based on Cross-modal Attention for Social Media Microblog Recognition

CDN: Collaborate Decision Network Based on Cross-modal Attention for Social Media Microblog Recognition

Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis

MSA-HCL: Multimodal sentiment analysis model with hybrid contrastive learning

Contact Info

Product

Resources

About