Targeted Aspect-Based Multimodal Sentiment Analysis: An Attention Capsule Extraction and Multi-Head Fusion Network

Gu, Donghong; Wang, Hua; Cai, Shaohua; Yang, Chi; Song, Zhengxin; Xiao, Luwei

doi:10.1109/access.2021.3126782

Cited by 31 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wang et al [41] designed an Attention Capsule Extraction and Multi-head Fusion Network (EF-Net) for MABSA, the basic framework of which is shown in Figure 5. EF-Net extracts image features using ResNet-152 and inputs them into a single layer capsule network to obtain the position information of the target in the image.…”

Section: ) Attention Mechanism-based Masc Methodsmentioning

confidence: 99%

A Survey on Multimodal Aspect-Based Sentiment Analysis

Zhao,

Yang,

Bai

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Multimodal Aspect-Based Sentiment Analysis (MABSA), as an emerging task in the field of sentiment analysis, has recently received widespread attention. Its aim is to combine relevant multimodal data to determine the sentiment polarity of a given aspect in text. Researchers have surveyed both aspectbased sentiment analysis and multimodal sentiment analysis, but, to the best of our knowledge, there is no survey on MABSA. Therefore, in order to assist related researchers to know MABSA better, we surveyed the research work on MABSA in recent years. Firstly, the relevant concepts of MABSA were introduced. Secondly, the existing research methods for the two subtasks of MABSA research (that is, multimodal aspect sentiment classification and aspect sentiment pairs extraction) were summarized and analyzed, and the advantages and disadvantages of each type of method were analyzed. Once again, the commonly used evaluation corpus and indicators for MABSA were summarized, and the evaluation results of existing research methods on the corpus were also compared. Finally, the possible research trends for MABSA were envisioned.INDEX TERMS Multimodal Aspect-Based Sentiment Analysis, multimodal aspect sentiment classification, aspect sentiment pairs extraction.

show abstract

Section: ) Attention Mechanism-based Masc Methodsmentioning

confidence: 99%

A Survey on Multimodal Aspect-Based Sentiment Analysis

Zhao,

Yang,

Bai

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Zhang et al (2021) [ 28 ] employed a pair of memory networks to capture intra-modal information and extract interaction information between different modalities, then designed a discriminative matrix to supervise the fusion of modal information. Gu et al (2021) [ 29 ] designed an Attention Capsule Extraction and Multi-Head Fusion Network for aspect-level multimodal sentiment classification. Through the integration of multi-head attention mechanisms and capsule networks, it captures interactions between multimodal inputs.…”

Section: Related Workmentioning

confidence: 99%

Joint analysis of interaction and psychological characteristics in english teaching based on multimodal integration

2024

BMC Psychol

View full text Add to dashboard Cite

The intersection of psychology and English teaching is profound, as the application of psychological principles not only guides specific English instruction but also elevates the overall quality of teaching. This paper takes a multimodal approach, incorporating image, acoustics, and text information, to construct a joint analysis model for English teaching interaction and psychological characteristics. The novel addition of an attention mechanism in the multimodal fusion process enables the development of an English teaching psychological characteristics recognition model. The initial step involves balancing the proportions of each emotion, followed by achieving multimodal alignment. In the cross-modal stage, the interaction of image, acoustics, and text is facilitated through a cross-modal attention mechanism. The utilization of a multi-attention mechanism not only enhances the network’s representation capabilities but also streamlines the complexity of the model. Empirical results demonstrate the model’s proficiency in accurately identifying five psychological characteristics. The proposed method achieves a classification accuracy of 90.40% for psychological features, with a commendable accuracy of 78.47% in multimodal classification. Furthermore, the incorporation of the attention mechanism in feature fusion contributes to an improved fusion effect.

show abstract

“…• EF-NET [41]: Uses a network based on Multi-Head Attention (MHA) and ResNet-152 to process text and images, respectively, and employs MHA to capture interactions between multimodal inputs.…”

Section: B Baselinesmentioning

confidence: 99%

Position Perceptive Multi-Hop Fusion Network for Multimodal Aspect-Based Sentiment Analysis

Fan,

Chen

2024

IEEE Access

View full text Add to dashboard Cite

With the increasing prevalence of multimodal user-generated content on social media, Multimodal Aspect-Based Sentiment Analysis (MABSA) has garnered significant attention in recent years. MABSA aims to classify the sentiment polarity of aspects mentioned in textual content by leveraging both textual and visual modalities. Previous studies have primarily focused on using transformer-based models to fuse information from different modalities. However, the presence of irrelevant information in images and text related to specific aspects can negatively impact sentiment analysis results. Many studies still face challenges in handling the noise introduced during the multimodal fusion process. To address this issue, we propose a novel Position-Perceptive Multi-hop Fusion Network (PPMFN) in this paper. Our method includes a position-perceptive module and an aspect-guided multi-hop interactive attention fusion module, aimed at enhancing modal position perception and aspect-guided multi-hop interactions to filter out irrelevant noise information. Typically, parts closer to the target are more relevant. The position-perceptive module is used to capture positional features from both images and text. The aspect-guided fusion module leverages a multihop interactive attention network to merge multimodal features while eliminating irrelevant information from both images and text. Extensive experimental results demonstrate that our framework consistently surpasses strong baseline models across two public datasets.

show abstract

Targeted Aspect-Based Multimodal Sentiment Analysis: An Attention Capsule Extraction and Multi-Head Fusion Network

Cited by 31 publications

References 19 publications

A Survey on Multimodal Aspect-Based Sentiment Analysis

A Survey on Multimodal Aspect-Based Sentiment Analysis

Joint analysis of interaction and psychological characteristics in english teaching based on multimodal integration

Position Perceptive Multi-Hop Fusion Network for Multimodal Aspect-Based Sentiment Analysis

Contact Info

Product

Resources

About