Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Abstract

Facial action unit detection has emerged as an important task within facial expression analysis, aimed at detecting specific pre-defined, objective facial expressions, such as lip tightening and cheek raising. This paper presents our submission to the Affective Behavior Analysis in-the-wild (ABAW) 2023 Competition for AU detection. We propose a multi-modal method for facial action unit detection with visual, acoustic, and lexical features extracted from the large pre-trained models. To provide high-quality details for visual feature extraction, we apply super-resolution and face alignment to the training data. Our approach achieves the F1 score of 52.3% on the official validation set.

Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detection

“…Moreover, we compare our results with those of ME-Graph [31], and our method outperforms theirs by an average F1-score of 3.1.These results demonstrate the effectiveness of our approach in detecting AUs. 51.0 CtyunAI [60] 48.9 HSE-NN-SberAI [39] 48.8 USTC-AC [51] 48.1 HFUT-MAC [59] 47.5 SCLAB-CNU [35] 45.6 USC-IHP [53] 42.9 Baseline [20] 36.5…”