A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Rajasekar, Gnana Praveen; Melo, Wheidima Carneiro de; Ullah, Nasib; Aslam, Haseeb; Zeeshan, Osama; Denorme, Théo; Pedersoli, Marco; Koerich, Alessandro L.; Cardinal, Patrick; Granger, Éric

doi:10.1109/cvprw56347.2022.00278

Cited by 48 publications

(13 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since multiple modes contain different information, it is an urgent problem to effectively capture the complementary relationship between different mode information. To combine different modes reliably, we introduce a cross‐attention fusion mechanism 50 to effectively mine the information between different modalities, while preserving the features intra‐modality. Specifically, we first extract the deep features which denote as

{X}_a\in {\mathrm{\mathbb{R}}}^{d_a\times L}

and

{X}_a\in {\mathrm{\mathbb{R}}}^{d_v\times L}

from different modes.…”

Section: The Proposed Methodsmentioning

confidence: 99%

A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

Xu,

Du,

Wang

et al. 2023

Computational Intelligence

View full text Add to dashboard Cite

Emotional recognition in conversations (ERC) is increasingly being applied in various IoT devices. Deep learning‐based multimodal ERC has achieved great success by leveraging diverse and complementary modalities. Although most existing methods try to adopt attention mechanisms to fuse different information, these methods ignore the complementarity between modalities. To this end, the joint cross‐attention model is introduced to alleviate this issue. However, multi‐scale feature information on different modalities is not utilized. Moreover, the context relationship plays an important role in feature extraction in the expression recognition task. In this paper, we propose a novel joint hierarchical graph convolution network (JHGCN) which exploits different layer features and context relationships for facial expression recognition based on audio‐visual (A‐V) information. Specifically, we adopt different deep networks to extract features from different modalities individually. For V modality, we construct V graph data based on patch embeddings which are extracted from the transformer encoder. Moreover, we embed the graph convolution which can leverage the intra‐modality relationships with the transformer encoder. Then, the deep feature from different layers is fed to the hierarchical fusion module to enhance feature representation. At last, we use the joint cross‐attention mechanism to exploit the complementary inter‐modality relationships. To validate the proposed model, we have conducted various experiments on the AffWild2 and CMU‐MOSI datasets. All results confirm that our proposed model achieves highly promising performance compared to the joint cross‐attention model and other methods.

show abstract

{X}_a\in {\mathrm{\mathbb{R}}}^{d_a\times L}

and

{X}_a\in {\mathrm{\mathbb{R}}}^{d_v\times L}

from different modes.…”

Section: The Proposed Methodsmentioning

confidence: 99%

A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

Xu,

Du,

Wang

et al. 2023

Computational Intelligence

View full text Add to dashboard Cite

show abstract

“…The self-attention is employed to model the intra-modal dependencies, while the cross-attention regards the representation of other modality as the query vector to capture the inter-model relationship. After that, we adopt a fusion method similar to [9] to further fuse two modalities.…”

Section: Attention-based Cross-modal Feature Fusionmentioning

confidence: 99%

“…Despite the above research progress, KWS in realistic environments with a low SNR is still challenging. Since signals from audio modality can be susceptible to contamination by the complex environment, multi-modal based methods have been well explored in various tasks, including emotion recognition [9], scene classification [10], etc., aiming to provide complementary information for the systems. These studies have proven that the introduction of visual modality can improve the robustness of systems.…”

Section: Introductionmentioning

confidence: 99%

VE-KWS: Visual Modality Enhanced End-to-End Keyword Spotting

Zhang¹,

Wang²,

Guo³

et al. 2023

Preprint

View full text Add to dashboard Cite

The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting crossattention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves 2.79% false rejection rate and 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge.

show abstract

“…Praveen et. al [35] improved the performance of the emotion recognition model by using a cross-attention module to fuse audio-visual features. Shriwardhana et.…”

Section: Related Workmentioning

confidence: 99%

Korean Drama Scene Transcript Dataset for Emotion Recognition in Conversations

et al. 2022

View full text Add to dashboard Cite

Understanding emotions in conversation is a challenging task as the sentences often have an implied meaning which is not generally understood in isolation. Efficient use of contextual information is important for emotion recognition in conversations. Many of the published datasets provide contextual information for situations such as text-based online messaging, chatbots, and movie dialogues. However, such dialogue-based datasets are collected by selecting the ideal conversational situations and thus do not include many variations in dialogue length and number of participants. Therefore, such datasets may not be applicable for emotion recognition in text-based movie transcripts, where scenes contain variations in the number of speakers and length of the spoken sentences. We present a conversation dataset based on the Korean TV show transcripts for analysis of the emotions in presence of scene context. Korean Drama Scene Transcript dataset for Emotion Recognition (KD-EmoR) is a text-based conversation dataset. We analyze three classes of complex emotions namely euphoria, dysphoria, and neutral in the scenes of TV Drama to build a publicly available dataset for further research. We developed a context-aware deep learning model to classify the emotions utilizing speaker-level context and scene context and achieved an F1 score of 0.63 on the proposed dataset.

show abstract

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Cited by 48 publications

References 79 publications

A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

VE-KWS: Visual Modality Enhanced End-to-End Keyword Spotting

Korean Drama Scene Transcript Dataset for Emotion Recognition in Conversations

Contact Info

Product

Resources

About