Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

Abstract

Multimodal emotion recognition has recently gained much attention since it can leverage diverse and complementary relationships over multiple modalities, such as audio, visual, and biosignals. Most state-of-the-art methods for audio-visual (A-V) fusion rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. In this paper, we focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. Specifically, we propose a joint cross-attention model that relies on the complementary relationships to extract the salient features across A-V modalities, allowing for accurate prediction of continuous values of valence and arousal. The proposed fusion model efficiently leverages the inter-modal relationships, while reducing the heterogeneity between features. In particular, it computes cross-attention weights based on the correlation between joint feature representations, and that of individual modalities. By deploying a joint A-V feature representation into the crossattention module, the performance of our fusion module improves significantly over the vanilla cross-attention module. Experimental results 1 on the AffWild2 dataset highlight the robustness of our proposed A-V fusion model. It has achieved a concordance correlation coefficient (CCC) of 0.374 (0.663) and 0.363 (0.584) for valence and arousal, respectively, on test set (validation set). This is a significant improvement over the baseline of third challenge of Affective Behavior Analysis in-the-wild (ABAW3) competition, with a CCC of 0.180 (0.310) and 0.170 (0.170).

Video-Based Cross-Modal Auxiliary Network for Multimodal Sentiment Analysis

“…Since multimodal analysis can leverage both independent and complementary information to provide comprehensive representations [25], such a technique has drawn much interest in sentiment analysis [26]. The multimodal fusion strategy can be grouped into feature-level fusion [27] and decision-level fusion, and the latter is the current mainstream fusion approach [28].…”

“…The video-based audiovisual fusion focuses more on capturing the complex spatiotemporal and semantic relationships among consecutive video frames. Under this condition, the Recurrent network with its variants and the attention mechanism [32] are widely used in multimodal fusion [26]. For example, Ou et al [4] extended the attention mechanism to obtain a global representation of effective video.…”

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

“…1. More specifically, we have considered four baseline models: Cross-Attention (CA) [2], Joint Cross-Attention (JCA) [12], Recursive Joint Cross-Attention (RJCA) [13], and Transformerbased Cross-Attention (TCA) [9]. First, we have implemented a simple baseline of the CA model to capture the complementary relationship between audio and visual modalities.…”

“…However, these methods fail to effectively capture the complementary intermodal relationships. Unlike these approaches, Praveen et al [2,12,13] explored CA to capture the complementary relationship across audio and visual modalities and showed improvement over prior methods. Although [2,12,13] achieved superior performance, they failed to deal with weak complementary relationships.…”

“…Recently, multimodal learning has achieved remarkable success by leveraging complementary relationships across the modalities using cross-modal attention for Emotion Recognition (ER) [1,2]. The idea of cross-modal attention, often termed as Cross-Attention (CA) or co-attention, is to leverage one modality to attend to another modality based on crossmodal interactions [3].…”