Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion

Zhao, Jian; Dong, Wenhua; Shi, Lijuan; Qiang, Wenqian; Kuang, Zhejun; Xu, Di; An, Tianbo

doi:10.3390/s22155528

Cited by 9 publications

(2 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This arrangement facilitates the capture of more nuanced unimodal, bimodal, and trimodal linkages. By examining and analyzing various feature integration methods for text and speech separately, a multimodal feature fusion approach [29] was suggested for imbalanced sample data. The aim of this proposal is to implement multimodal emotion recognition effectively.…”

Section: Attention Based Cross-modal Emotion Recognitionmentioning

confidence: 99%

Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures

Makhmudov,

Kultimuratov,

Cho

2024

Applied Sciences

View full text Add to dashboard Cite

Emotion detection holds significant importance in facilitating human–computer interaction, enhancing the depth of engagement. By integrating this capability, we pave the way for forthcoming AI technologies to possess a blend of cognitive and emotional understanding, bridging the divide between machine functionality and human emotional complexity. This progress has the potential to reshape how machines perceive and respond to human emotions, ushering in an era of empathetic and intuitive artificial systems. The primary research challenge involves developing models that can accurately interpret and analyze emotions from both auditory and textual data, whereby auditory data require optimizing CNNs to detect subtle and intense emotional fluctuations in speech, and textual data necessitate access to large, diverse datasets to effectively capture nuanced emotional cues in written language. This paper introduces a novel approach to multimodal emotion recognition, seamlessly integrating speech and text modalities to accurately infer emotional states. Employing CNNs, we meticulously analyze speech using Mel spectrograms, while a BERT-based model processes the textual component, leveraging its bidirectional layers to enable profound semantic comprehension. The outputs from both modalities are combined using an attention-based fusion mechanism that optimally weighs their contributions. The proposed method here undergoes meticulous testing on two distinct datasets: Carnegie Mellon University’s Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset and the Multimodal Emotion Lines Dataset (MELD). The results demonstrate superior efficacy compared to existing frameworks, achieving an accuracy of 88.4% and an F1-score of 87.9% on the CMU-MOSEI dataset, and a notable weighted accuracy (WA) of 67.81% and a weighted F1 (WF1) score of 66.32% on the MELD dataset. This comprehensive system offers precise emotion detection and introduces several significant advancements in the field.

show abstract

Section: Attention Based Cross-modal Emotion Recognitionmentioning

confidence: 99%

Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures

Makhmudov,

Kultimuratov,

Cho

2024

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…This indicates that titles are a powerful sentimental differentiator between fake and real news. Fine-grained sentiment analysis of subjective content can be approached positively thanks to studies in the field [ 65 ]. Sentiment analysis usually operates at a coarser level in research efforts.…”

Section: Background Of Studymentioning

confidence: 99%

Fake News Detection Model on Social Media by Leveraging Sentiment Analysis of News Content and Emotion Analysis of Users’ Comments

Hamed

Aziz

Yaakub

2023

Sensors

View full text Add to dashboard Cite

Nowadays, social media has become the main source of news around the world. The spread of fake news on social networks has become a serious global issue, damaging many aspects, such as political, economic, and social aspects, and negatively affecting the lives of citizens. Fake news often carries negative sentiments, and the public’s response to it carries the emotions of surprise, fear, and disgust. In this article, we extracted features based on sentiment analysis of news articles and emotion analysis of users’ comments regarding this news. These features were fed, along with the content feature of the news, to the proposed bidirectional long short-term memory model to detect fake news. We used the standard Fakeddit dataset that contains news titles and comments posted regarding them to train and test the proposed model. The suggested model, using extracted features, provided a high detection accuracy of 96.77% of the Area under the ROC Curve measure, which is higher than what other state-of-the-art studies offer. The results prove that the features extracted based on sentiment analysis of news, which represents the publisher’s stance, and emotion analysis of comments, which represent the crowd’s stance, contribute to raising the efficiency of the detection model.

show abstract

Compact bilinear pooling and multi-loss network for social media multimodal classification

Li,

Zheng,

Zhu

et al. 2024

SIViP

View full text Add to dashboard Cite

Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion

Cited by 9 publications

References 27 publications

Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures

Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures

Fake News Detection Model on Social Media by Leveraging Sentiment Analysis of News Content and Emotion Analysis of Users’ Comments

Compact bilinear pooling and multi-loss network for social media multimodal classification

Contact Info

Product

Resources

About