Facial Action Unit Recognition With Multi-models Ensembling

Wu, Yizhe; Fengsheng, Qiao,; Meng, Liyu; Deng, Yuanyuan; Liu, Chuanhe

doi:10.48550/arxiv.2203.13046

Cited by 3 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Training. The learning objective is the sum of the weighted BCE loss and weighted multi-label loss * proposed by Jiang et al [9] which is the second place of the 3rd ABAW challenge [12]. where W 1 and W 2 are the loss weights for each AU.…”

Section: Training and Inferencementioning

confidence: 99%

See 1 more Smart Citation

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Yin¹,

Tran²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Facial action unit detection has emerged as an important task within facial expression analysis, aimed at detecting specific pre-defined, objective facial expressions, such as lip tightening and cheek raising. This paper presents our submission to the Affective Behavior Analysis in-the-wild (ABAW) 2023 Competition for AU detection. We propose a multi-modal method for facial action unit detection with visual, acoustic, and lexical features extracted from the large pre-trained models. To provide high-quality details for visual feature extraction, we apply super-resolution and face alignment to the training data. Our approach achieves the F1 score of 52.3% on the official validation set.

show abstract

Section: Training and Inferencementioning

confidence: 99%

“…The model is evaluated on our validation set at the end of every epoch. Following Jiang et al [9], the loss weights of the BCE loss is…”

Section: Implementation and Training Detailsmentioning

confidence: 99%

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Yin¹,

Tran²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the previous editions of the ABAW competition, many teams utilized multimodal features (Kollias 2022;Jiang et al 2022;Jin et al 2021;Meng et al 2022;Zhang et al 2022b,a).The model proposed by Meng et al(Meng et al 2022) leverages both audio and visual features, ultimately achieving first place in the VA track . To fully exploit the in-the-wild emotion information, Zhang et al (Zhang et al 2022b) utilizes the multimodal information from the images, audio and text and propose a unified multimodal framework for AU detection and expression recognition.The proposed framework achieved the highest scores on both tasks.…”

Section: Related Work Multimodal Featuresmentioning

confidence: 99%

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Zhang¹,

An²,

Zishun³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we present our solutions for the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW), which includes four sub-challenges of Valence-Arousal (VA) Estimation, Expression (Expr) Classification, Action Unit (AU) Detection and Emotional Reaction Intensity (ERI) Estimation. The 5th ABAW competition focuses on facial affect recognition utilizing different modalities and datasets. In our work, we extract powerful audio and visual features using a large number of sota models. These features are fused by Transformer Encoder and TEMMA. Besides, to avoid the possible impact of large dimensional differences between various features, we design an Affine Module to align different features to the same dimension. Extensive experiments demonstrate that the superiority of the proposed method. For the VA Estimation sub-challenge, our method obtains the mean Concordance Correlation Coefficient (CCC) of 0.6066. For the Expression Classification subchallenge, the average F1 Score is 0.4055. For the AU Detection sub-challenge, the average F1 Score is 0.5296. For the Emotional Reaction Intensity Estimation sub-challenge, the average pearson's correlations coefficient on the validation set is 0.3968. All of the results of four sub-challenges outperform the baseline with a large margin.

show abstract

“…AU detection involves the identification and tracking of subtle facial movements that correspond to specific emotional states or expressions. The improved IResnet100 introduced by Jiang et.al [8] was utilized to address the AU detection task. And Wang et.al [27] proposed a action units correlation module to learn relationships between each AU labels and proved the effectiveness of the method.…”

Section: Au Detectionmentioning

confidence: 99%

Spatial-temporal Transformer for Affective Behavior Analysis

Zou¹,

Wang²,

Wen³

et al. 2023

Preprint

View full text Add to dashboard Cite

The in-the-wild affective behavior analysis has been an important study. In this paper, we submit our solutions for the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW), which includes V-A Estimation, Facial Expression Classification and AU Detection Sub-challenges. We propose a Transformer Encoder with Multi-Head Attention framework to learn the distribution of both the spatial and temporal features. Besides, there are virious effective data augmentation strategies employed to alleviate the problems of sample imbalance during model training. The results fully demonstrate the effectiveness of our proposed model based on the Aff-Wild2 dataset.

show abstract

Facial Action Unit Recognition With Multi-models Ensembling

Cited by 3 publications

References 15 publications

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Spatial-temporal Transformer for Affective Behavior Analysis

Contact Info

Product

Resources

About