End-to-End Learning for Multimodal Emotion Recognition in Video With Adaptive Loss

Huynh, Van Thong; Yang, Hyung-Jeong; Lee, Guee-Sang; Kim, Soo-Hyung

doi:10.1109/mmul.2021.3080305

Cited by 6 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For all emotion classes and on overall, our method achieves much better results than CAE-LR [25], showing the effectiveness of the contrastive loss in multimodal setting compared to convolutional autoencoders. It is worth noting that, on average, our method is better than several fully supervised techniques: MESM [21], FE2E [21], Graph-MFN [43], [51], CIA [46]. Considering that these methods integrate relatively complex supervised techniques; attention mechanisms, transformers, graphs, the better performance of our method is very promising.…”

Section: Comparisons With the State-of-the-art Methodsmentioning

confidence: 84%

“…There are a lot of attempts applying end-to-end learning [27], [26], [50], [51], but only [21] compared a fully end-toend method (defined as jointly optimizing feature extraction and feature learning stages [21]) with the two-phase pipelines (i.e., feature extraction is independent from multimodal learning). Indeed, it is very common in the MER litreature to apply the feature extraction step separately.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Franceschini¹,

Fini²,

Beyan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Emotion recognition is involved in several real-world applications. With an increase in available modalities, automatic understanding of emotions is being performed more accurately. The success in Multimodal Emotion Recognition (MER), primarily relies on the supervised learning paradigm. However, data annotation is expensive, time-consuming, and as emotion expression and perception depends on several factors (e.g., age, gender, culture) obtaining labels with a high reliability is hard. Motivated by these, we focus on unsupervised feature learning for MER. We consider discrete emotions, and as modalities text, audio and vision are used. Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature. Our end-to-end feature learning approach has several differences (and advantages) compared to existing MER methods: i) it is unsupervised, so the learning is lack of data labelling cost; ii) it does not require data spatial augmentation, modality alignment, large number of batch size or epochs; iii) it applies data fusion only at inference; and iv) it does not require backbones pre-trained on emotion recognition task. The experiments on benchmark datasets show that our method outperforms several baseline approaches and unsupervised learning methods applied in MER. Particularly, it even surpasses a few supervised MER state-of-the-art.

show abstract

Section: Comparisons With the State-of-the-art Methodsmentioning

confidence: 84%

Section: Related Workmentioning

confidence: 99%

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Franceschini¹,

Fini²,

Beyan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…As large models gain prominence, word embeddings and pre-trained models have seen significant success in sentiment analysis. Word2Vec [24] maps semantically similar words to similar vector spaces, while GloVe [25] derives semantic relationships between words based on global co-occurrence. ELMo [26] introduces context-aware embeddings, allowing word representations to vary based on their specific contexts within sentences.…”

Section: Related Workmentioning

confidence: 99%

Cross-Modal Sentiment Analysis Based on CLIP Image-Text Attention Interaction

Lu,

Ni,

Ding

2024

IJACSA

View full text Add to dashboard Cite

Multimodal sentiment analysis is a traditional textbased sentiment analysis technique. However, the field of multimodal sentiment analysis still faces challenges such as inconsistent cross-modal feature information, poor interaction capabilities, and insufficient feature fusion. To address these issues, this paper proposes a cross-modal sentiment model based on CLIP image-text attention interaction. The model utilizes pre-trained ResNet50 and RoBERTa to extract primary image-text features. After contrastive learning with the CLIP model, it employs a multi-head attention mechanism for cross-modal feature interaction to enhance information exchange between different modalities. Subsequently, a cross-modal gating module is used to fuse feature networks, combining features at different levels while controlling feature weights. The final output is fed into a fully connected layer for sentiment recognition. Comparative experiments are conducted on the publicly available datasets MSVA-Single and MSVA-Multiple. The experimental results demonstrate that our model achieved accuracy rates of 75.38% and 73.95% , and F1-scores of 75.21% and 73.83% on the mentioned datasets, respectively. This indicates that the proposed approach exhibits higher generalization and robustness compared to existing sentiment analysis models.

show abstract

Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

Halawa,

Blume,

Bideau

et al. 2024

2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

End-to-End Learning for Multimodal Emotion Recognition in Video With Adaptive Loss

Cited by 6 publications

References 18 publications

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Cross-Modal Sentiment Analysis Based on CLIP Image-Text Attention Interaction

Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

Contact Info

Product

Resources

About