A Multitask Learning Framework for Multimodal Sentiment Analysis

Jiang, Dazhi; Wei, Runguo; Líu, Hao; Wen, Jintao; Tu, Geng; Zheng, Lin; Wang, Zhaoxia

doi:10.1109/icdmw53433.2021.00025

Cited by 11 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This module aids in the integration of textual information into the model. The selfattention mechanism [29,30] involves the creation of three matrices, namely Q (query), K (key), and V (value), by multiplying the matrix H with weight matrices W Q , W K , W V . These weight matrices are trained jointly in the self-attention mechanism.…”

Section: Multi-head Attention Fusion Modulementioning

confidence: 99%

Multi-Modal Sarcasm Detection with Sentiment Word Embedding

Fu,

Liu,

Wang

et al. 2024

Electronics

Self Cite

View full text Add to dashboard Cite

Sarcasm poses a significant challenge for detection due to its unique linguistic phenomenon where the intended meaning is often opposite of the literal expression. Current sarcasm detection technology primarily utilizes multi-modal processing, but the connotative semantic information provided by the modality itself is limited. It is a challenge to mine the semantic information contained in the combination of sarcasm samples and external commonsense knowledge. Furthermore, as the essence of sarcasm detection lies in measuring emotional inconsistency, the rich semantic information may introduce excessive noise to inconsistency measurement. To mitigate these limitations, we propose a hierarchical framework in this paper. Specifically, to enrich the semantic information of each modality, our approach uses sentiment dictionaries to obtain the sentiment vectors by evaluating the words extracted from various modalities, and then combines them with each modality. Furthermore, in order to mine the joint semantic information implied in the modalities and improve measurement of emotional inconsistency, the emotional information representation obtained by fusing each modality’s data is concatenated with the sentiment vector. Then, cross-modal fusion is performed through cross-attention, and, finally, the sarcasm is recognized by fusing low-level information in the cross-modal fusion layer. Our model is evaluated on a public multi-modal sarcasm detection dataset based on Twitter, and the results demonstrate its superiority.

show abstract

Section: Multi-head Attention Fusion Modulementioning

confidence: 99%

Multi-Modal Sarcasm Detection with Sentiment Word Embedding

Fu,

Liu,

Wang

et al. 2024

Electronics

Self Cite

View full text Add to dashboard Cite

show abstract

“…The soft labels contain more inter-class information (Hinton et al 2015), and the accuracy is further improved through combination with a multi-task learning strategy (Liu et al 2018, Jiang et al 2021. Different grades of sacroiliitis have specific features (Linden et al 1984).…”

Section: Loss Functions Of Multi-task Classificationmentioning

confidence: 99%

Sacroiliitis diagnosis based on interpretable features and multi-task learning

Liu,

Zhang,

Zhang

et al. 2024

Phys. Med. Biol.

View full text Add to dashboard Cite

Objective. Sacroiliitis is an early pathological manifestation of ankylosing spondylitis (AS), and a positive sacroiliitis test on imaging may help clinical practitioners diagnose AS early. Deep learning based automatic diagnosis algorithms can deliver grading findings for sacroiliitis, however, it requires a large amount of data with precise labels to train the model and lacks grading features visualization. In this paper, we aimed to propose a radiomics and deep learning based deep feature visualization positive diagnosis algorithm for sacroiliitis on CT scans. Visualization of grading features can enhance clinical interpretability with visual grading features, which assist doctors in diagnosis and treatment more effectively.Approach. The region of interest (ROI) is identified by segmenting the sacroiliac joint (SIJ) 3D CT images using a combination of the U-net model and certain statistical approaches. Then, in addition to extracting spatial and frequency domain features from ROI according to the radiographic manifestations of sacroiliitis, the radiomics features have also been integrated into the proposed encoder module to obtain a powerful encoder and extract features effectively. Finally, a multi-task learning technique and five-class labels are utilized to help with performing positive tests to reduce discrepancies in the evaluation of several radiologists.Main results. On our private dataset, proposed methods have obtained an accuracy rate of 87.3$\%$, which is 9.8$\%$ higher than the baseline and consistent with assessments made by qualified medical professionals.Significance. The results of the ablation experiment and interpreting analysis demonstrated that the proposed methods are applied in automatic CT scan sacroiliitis diagnosis due to their excellently interpretable and portable advantages.

show abstract

“…Applications such as social networks and news websites have generated abundant multimodal data. As a result, people have conducted extensive multimodal investigations; for example, sentiment analysis [23][24][25][26][27], image and text retrieval [28], reason extraction [29,30], and sarcasm detection [31]. Unlike text-modal-based sarcasm detection, multimodal sarcasm detection aims to identify sarcastic expressions implied in multimodal data.…”

Section: Multimodal Sarcasm Detectionmentioning

confidence: 99%

A Multi-View Interactive Approach for Multimodal Sarcasm Detection in Social Internet of Things with Knowledge Enhancement

Liu,

Yang,

2024

Applied Sciences

Self Cite

View full text Add to dashboard Cite

Multimodal sarcasm detection is a developing research field in social Internet of Things, which is the foundation of artificial intelligence and human psychology research. Sarcastic comments issued on social media often imply people’s real attitudes toward the events they are commenting on, reflecting their current emotional and psychological state. Additionally, the limited memory of Internet of Things mobile devices has posed challenges in deploying sarcastic detection models. An abundance of parameters also leads to an increase in the model’s inference time. Social networking platforms such as Twitter and WeChat have generated a large amount of multimodal data. Compared to unimodal data, multimodal data can provide more comprehensive information. Therefore, when studying sarcasm detection on social Internet of Things, it is necessary to simultaneously consider the inter-modal interaction and the number of model parameters. In this paper, we propose a lightweight multimodal interaction model with knowledge enhancement based on deep learning. By integrating visual commonsense knowledge into the sarcasm detection model, we can enrich the semantic information of image and text modal representation. Additionally, we develop a multi-view interaction method to facilitate the interaction between modalities from different modal perspectives. The experimental results indicate that the model proposed in this paper outperforms the unimodal baselines. Compared to multimodal baselines, it also has similar performance with a small number of parameters.

show abstract

A Multitask Learning Framework for Multimodal Sentiment Analysis

Cited by 11 publications

References 27 publications

Multi-Modal Sarcasm Detection with Sentiment Word Embedding

Multi-Modal Sarcasm Detection with Sentiment Word Embedding

Sacroiliitis diagnosis based on interpretable features and multi-task learning

A Multi-View Interactive Approach for Multimodal Sarcasm Detection in Social Internet of Things with Knowledge Enhancement

Contact Info

Product

Resources

About