Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Abstract

In this paper, we present our solutions for the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW), which includes four sub-challenges of Valence-Arousal (VA) Estimation, Expression (Expr) Classification, Action Unit (AU) Detection and Emotional Reaction Intensity (ERI) Estimation. The 5th ABAW competition focuses on facial affect recognition utilizing different modalities and datasets. In our work, we extract powerful audio and visual features using a large number of sota models. These features are fused by Transformer Encoder and TEMMA. Besides, to avoid the possible impact of large dimensional differences between various features, we design an Affine Module to align different features to the same dimension. Extensive experiments demonstrate that the superiority of the proposed method. For the VA Estimation sub-challenge, our method obtains the mean Concordance Correlation Coefficient (CCC) of 0.6066. For the Expression Classification subchallenge, the average F1 Score is 0.4055. For the AU Detection sub-challenge, the average F1 Score is 0.5296. For the Emotional Reaction Intensity Estimation sub-challenge, the average pearson's correlations coefficient on the validation set is 0.3968. All of the results of four sub-challenges outperform the baseline with a large margin.

Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detection

“…Moreover, we compare our results with those of ME-Graph [31], and our method outperforms theirs by an average F1-score of 3.1.These results demonstrate the effectiveness of our approach in detecting AUs. 51.0 CtyunAI [60] 48.9 HSE-NN-SberAI [39] 48.8 USTC-AC [51] 48.1 HFUT-MAC [59] 47.5 SCLAB-CNU [35] 45.6 USC-IHP [53] 42.9 Baseline [20] 36.5…”

Emotion Recognition beyond Pixels: Leveraging Facial Point Landmark Meshes

“…Achieving a 53.07% F1-score, while not considered high, is still acceptable and reasonable in this type of application [44,[46][47][48][49][50]. The complexity and inherent ambiguity of emotion recognition, coupled with the dataset's representation, make it challenging to achieve notable performance.…”

“…F1-Score Yu et al [46] 0.3075 Xue et al [47] 0.3218 Savchenko [48] 0.3292 Zhang et al [49] 0.3337 Zhou et al [50] 0.3532 Proposed approach 0.5307 * Value in bold represents the best performance.…”

“…This approach achieved an F1-score of 0.3292 [44]. A fused transformer encoder model using audio-visual input with an affine module was implemented in [49], reaching an F1-score of 0.3337 [44]. In [50], a multi-modal fusion model was developed.…”