Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis

Li, Jia; Zhang, Ziyang; Lang, Junjie; Jiang, Yueqi; An, Liuwei; Zou, Peng; Xu, Yangyang; Gao, Sheng; Lin, Jie; Fan, Chunxiao; Sun, Xiao; Wang, Meng

doi:10.1145/3551876.3554809

Cited by 28 publications

(17 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As a fusion of both, Hybrid fusion combines the advantages of both early fusion and late fusion. Li et al (Li et al 2022) proposes a hybrid fusion method which leads them to the winner of MuSe-Reaction competition. In (Li et al 2022), audio features , facial expression features and paragraph-level text embeddings are fused at the feature level and then fed into the MMA module to extract complementary information from different modalities and calculate interactions between modalities.…”

Section: Multimodal Fusionmentioning

confidence: 99%

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Zhang¹,

An²,

Zishun³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we present our solutions for the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW), which includes four sub-challenges of Valence-Arousal (VA) Estimation, Expression (Expr) Classification, Action Unit (AU) Detection and Emotional Reaction Intensity (ERI) Estimation. The 5th ABAW competition focuses on facial affect recognition utilizing different modalities and datasets. In our work, we extract powerful audio and visual features using a large number of sota models. These features are fused by Transformer Encoder and TEMMA. Besides, to avoid the possible impact of large dimensional differences between various features, we design an Affine Module to align different features to the same dimension. Extensive experiments demonstrate that the superiority of the proposed method. For the VA Estimation sub-challenge, our method obtains the mean Concordance Correlation Coefficient (CCC) of 0.6066. For the Expression Classification subchallenge, the average F1 Score is 0.4055. For the AU Detection sub-challenge, the average F1 Score is 0.5296. For the Emotional Reaction Intensity Estimation sub-challenge, the average pearson's correlations coefficient on the validation set is 0.3968. All of the results of four sub-challenges outperform the baseline with a large margin.

show abstract

Section: Multimodal Fusionmentioning

confidence: 99%

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Zhang¹,

An²,

Zishun³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Participants could utilise the text, audio, and video modality. Several systems were proposed to tackle this task, all of them employing the Transformer architecture [49,50,51].…”

Section: Multimodal Humour Recognitionmentioning

confidence: 99%

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Christ¹,

Amiriparian²,

Kathan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Humour is a substantial element of human affect and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection are solely based on staged data making them inadequate for 'real-world' applications. We address this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among various subjects, highlighting the individuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. Finally, we make our code publicly available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset is available upon request.

show abstract

“…The TEMMA model proposed by Li [20] at Muse 2022 won the championship in this challenge. Resnet-18 feature and DeepSpectrum feature were used in [20].…”

Section: Emotional Reaction Intensitymentioning

confidence: 99%

“…The TEMMA model proposed by Li [20] at Muse 2022 won the championship in this challenge. Resnet-18 feature and DeepSpectrum feature were used in [20]. The researchers [20] use a Resnet-18 model to extract static visual features from the videos, and a DeepSpectrum [2] model to extract audio features from the videos.…”

Section: Emotional Reaction Intensitymentioning

confidence: 99%

See 1 more Smart Citation

Hyperspectral Image Classification for Foreign Bodies in Medicine Based on Spatial-Spectral Fusion Feature

Zhu

Yin

et al. 2021

2021 China Automation Congress (CAC)

View full text Add to dashboard Cite

In this paper, we present our advanced solutions to the two sub-challenges of Affective Behavior Analysis in the wild (ABAW) 2023: the Emotional Reaction Intensity (ERI) Estimation Challenge and Expression (Expr) Classification Challenge. ABAW 2023 aims to tackle the challenge of affective behavior analysis in natural contexts, with the ultimate goal of creating intelligent machines and robots that possess the ability to comprehend human emotions, feelings, and behaviors. For the Expression Classification Challenge, we propose a streamlined approach that handles the challenges of classification effectively. However, our main contribution lies in our use of diverse models and tools to extract multimodal features such as audio and video cues from the Hume-Reaction dataset. By studying, analyzing, and combining these features, we significantly enhance the model's accuracy for sentiment prediction in a multimodal context. Furthermore, our method achieves outstanding results on the Emotional Reaction Intensity (ERI) Estimation Challenge, surpassing the baseline method by an impressive 84% increase, as measured by the Pearson Coefficient, on the validation dataset.

show abstract

Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis

Cited by 28 publications

References 22 publications

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Hyperspectral Image Classification for Foreign Bodies in Medicine Based on Spatial-Spectral Fusion Feature

Contact Info

Product

Resources

About