Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Yan, Xueming; Xue, Haiwei; Jiang, Shengyi; Liu, Ziang

doi:10.1080/08839514.2021.2000688

Cited by 27 publications

(5 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Presenting an innovative multimodal SA system, the study [28] meticulously integrates text-and image-based components, utilizing deep learning for feature learning and classification. In the literature [29], researchers introduced a novel approach, utilizing a cross-modal modeling multitensor fusion network for effective emotion fusion in multimodal SA. Regression and classification experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets.…”

Section: Research On Multimodal Sentiment Analysismentioning

confidence: 99%

Design and Efficacy of a Data Lake Architecture for Multimodal Emotion Feature Extraction in Social Media

Fan,

2024

IET Software

View full text Add to dashboard Cite

In the rapidly evolving landscape of social media, the demand for precise sentiment analysis (SA) on multimodal data has become increasingly pivotal. This paper introduces a sophisticated data lake architecture tailored for efficient multimodal emotion feature extraction, addressing the challenges posed by diverse data types. The proposed framework encompasses a robust storage solution and an innovative SA model, multilevel spatial attention fusion (MLSAF), adept at handling text and visual data concurrently. The data lake architecture comprises five layers, facilitating real-time and offline data collection, storage, processing, standardized interface services, and data mining analysis. The MLSAF model, integrated into the data lake architecture, utilizes a novel approach to SA. It employs a text-guided spatial attention mechanism, fusing textual and visual features to discern subtle emotional interplays. The model’s end-to-end learning approach and attention modules contribute to its efficacy in capturing nuanced sentiment expressions. Empirical evaluations on established multimodal sentiment datasets, MVSA-Single and MVSA-Multi, validate the proposed methodology’s effectiveness. Comparative analyses with state-of-the-art models showcase the superior performance of our approach, with an accuracy improvement of 6% on MVSA-Single and 1.6% on MVSA-Multi. This research significantly contributes to optimizing SA in social media data by offering a versatile and potent framework for data management and analysis. The integration of MLSAF with a scalable data lake architecture presents a strategic innovation poised to navigate the evolving complexities of social media data analytics.

show abstract

Section: Research On Multimodal Sentiment Analysismentioning

confidence: 99%

Design and Efficacy of a Data Lake Architecture for Multimodal Emotion Feature Extraction in Social Media

Fan,

2024

IET Software

View full text Add to dashboard Cite

show abstract

“…Yan et al [24] have presented Multimodal SA Utilizing MTFN by Cross-modal Modeling. In the presented method suggests a little swift growth of social media platforms, an increasing number of people are using videos on the internet to express their emotions and opinions.…”

Section: Literature Reviewmentioning

confidence: 99%

Application of Multimodal Data Fusion Attentive Dual Residual Generative Adversarial Network in Sentiment Recognition and Sentiment Analysis

Yongfang Zhang

2024

jes

View full text Add to dashboard Cite

Recent advancements in Internet technology have led to increased multi-modal data posting on social media, online shopping portals, and video repositories recognizing significance of inter-modal utterances before combining multiple modes. In this manuscript, Application of Multimodal Data Fusion Attentive Dual Residual Generative Adversarial Network in Sentiment Recognition and Sentiment Analysis (MDF-DRGAN-SR-SA) is proposed. The input data are collected from CMU-MOSI dataset. Initially the input data is preprocessed using Subaperture Keystone Transform Matched Filtering (SAKTMF) is used to clean unwanted data. Then, feature extraction is done by Two-Sided Offset Quaternion Linear Canonical Transform (TSOQLCT) to extract unimodal features likes acoustic, textual, visual. Then the selected features are given to ADRGAN classifying Sentiment Recognition and Sentiment Analysis likes positive, negative, neutral. In general, ADRGAN doesn’t express some adaption of optimization strategies for determining optimal parameters to assure accurate classification of Sentiment Recognition and Sentiment Analysis. Hence, Northern Goshawk Optimization Algorithm (GOA) is proposed to enhance weight parameter of ADRGAN, which precisely classifies the Sentiment Recognition and Sentiment Analysis in positive, negative and neutral. The proposed model is implemented and its efficiency is evaluated utilizing some performance metrics likes accuracy, precision, specificity, sensitivity,F1-score. The MDF-DRGAN-SR-SA method provides 25.85%, 26.79% and 27.63% higher accuracy; 35.66%, 34.97% and 26.57% higher precision; 28.18%, 29.52% and 25.68% higher specificity is compared with existing method such as Two-Level Multimodal Fusion for SA in Public Security (TMDF-SA-PS), Multimodal SA Depend on Adaptive Modality-Specific Weight Fusion Network (MFN-SA-AMW) and Multimodal SA Utilizing Multi-tensor Fusion Network and Cross-modal Modeling(MTFN-SA) respectively.

show abstract

“…Multimodal sentiment analysis is a series of tasks to analyze and predict information from multiple modalities i n a unified way. YanXueming et al [3] proposed a multi-tensor fusion network with cross-modal modeling. L W ang [4] a cross-modal hierarchical fusion approach for multimodal sentiment analysis is proposed.…”

Section: Related Workmentioning

confidence: 99%

Multimodal sentiment analysis of heterogeneous bimodal interactions

Wang,

Wei,

2024

Third International Conference on Electronic Information Engineering and Data Processing (EIEDP 2024)

View full text Add to dashboard Cite

Aiming at the problems of unsatisfactory cross-modal feature interaction in multimodal sentiment analysis and the influence of each modality on the final sentiment classification results. Through the research on multimodal sentiment analysis methods, a multimodal sentiment analysis model based on heterogeneous extracted feature network and bimodal interaction is constructed. The model firstly uses DPCNN network to capture the deep semantics of text, BiLSTM network to obtain the upper and lower dependencies of speech, and TCN network to extract the relevant features for facial images; in order to learn the fusion features between different modalities, the interaction information within and between modalities is captured by the bimodal interaction network layer; and then the attention mechanism is used to determine the attentional weights of the modalities in the final sentiment features, and to classify the influence of modalities on the final sentiment classification results. attentional weights and fuses the modalities; finally, the final sentiment classification is performed through the fully connected layer and multi-head attention. The model is experimented and evaluated on the datasets CMU-MOSI and CMU-MOSEI, and the experimental results show that the model outperforms other models in the sentiment analysis problem compared to the existing models.

show abstract

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Cited by 27 publications

References 24 publications

Design and Efficacy of a Data Lake Architecture for Multimodal Emotion Feature Extraction in Social Media

Design and Efficacy of a Data Lake Architecture for Multimodal Emotion Feature Extraction in Social Media

Application of Multimodal Data Fusion Attentive Dual Residual Generative Adversarial Network in Sentiment Recognition and Sentiment Analysis

Multimodal sentiment analysis of heterogeneous bimodal interactions

Contact Info

Product

Resources

About