Affective Behaviour Analysis Using Pretrained Model with Facial Prior

Li, Yifan; Sun, Haomiao; Zhaori, Liu,; Hu, Han; Shan, Shiguang

doi:10.1007/978-3-031-25075-0_2

Cited by 7 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It also used MAE pre-trained weights to enhance its performance. Li et al [23] also use MAE pre-trained weights combined with AffectNet supervised pre-trained weights and ranked 2nd in ABAW4. Zhang et al [37] proposed a transformer-based fusion module to fuse multi-modality features from audio, image, and word information.…”

Section: Related Workmentioning

confidence: 99%

Exploring Expression-related Self-supervised Learning for Affective Behaviour Analysis

Xue¹,

Yang²

2023

Preprint

View full text Add to dashboard Cite

This paper explores an expression-related selfsupervised learning (SSL) method (ContraWarping) to perform expression classification in the 5th Affective Behavior Analysis in-the-wild (ABAW) competition. Affective datasets are expensive to annotate, and SSL methods could learn from large-scale unlabeled data, which is more suitable for this task. By evaluating on the Aff-Wild2 dataset, we demonstrate that Con-traWarping outperforms most existing supervised methods and shows great application potential in the affective analysis area. Codes will be released on: https : //github.com/youqingxiaozhua/ABAW5.

show abstract

Section: Related Workmentioning

confidence: 99%

Exploring Expression-related Self-supervised Learning for Affective Behaviour Analysis

Xue¹,

Yang²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The top results have been achieved by the Masked Auto-Encoder (MAE) pretrained on unlabeled face images. For example, the EMMA ensemble of pre-trained MAE ViT (Vision Transformer) and CNN took the 2nd place [20]. The winner [37] adopted ensembles of various temporal encoders, multi-task frameworks, and unsupervised (MAE-based) and supervised (IResNet/DenseNet-based) visual feature representation learning.…”

Section: Related Workmentioning

confidence: 99%

“…As one can see, smoothing does not work for AU detection but can significantly improve the results for other tasks: up to 0.06 difference in F1-score for EXPR classification and up to 0.06 difference in mean CCC for VA prediction. Moreover, the smoothing works nicely even for blending the best ---0.981 SMMEmotionNet [23] 0.3648 0.2617 0.4737 1.1002 Two-Aspect Information Interaction [31] 0.515 0.207 0.385 1.107 SS-MFAR [4] 0.397 0.235 0.493 1.125 EfficientNet-B2 [27] 0.384 0.302 0.461 1.147 MAE+ViT [20] 0.4588 0.3028 0.5054 1.2671 Cross-attentive module [23] 0.499 0.333 0.456 1.288 MT-EmotiEffNet + OpenFace [29] 0.447 0.357 0.496 1.300 MAE+Transformer [37] 0 frame-level 0.4847 0.3578 0.5194 1.3619 + MT-EmotiDDAMFN smoothing, tAU = 0.5 0.5578 0.4168 0.5194 1.4939 models (Fig. 5).…”

Section: Multi-task Learning Challengementioning

confidence: 99%

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Savchenko¹

2021

2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY)

120

View full text Add to dashboard Cite

In this paper, we describe the results of the HSEmotion team in two tasks of the seventh Affective Behavior Analysis in-thewild (ABAW) competition, namely, multi-task learning for simultaneous prediction of facial expression, valence, arousal, and detection of action units, and compound expression recognition. We propose an efficient pipeline based on frame-level facial feature extractors pre-trained in multi-task settings to estimate valence-arousal and basic facial expressions given a facial photo. We ensure the privacy-awareness of our techniques by using the lightweight architectures of neural networks, such as MT-EmotiDDAMFN, MT-EmotiEffNet, and MT-EmotiMobileFaceNet, that can run even on a mobile device without the need to send facial video to a remote server. It was demonstrated that a significant step in improving the overall accuracy is the smoothing of neural network output scores using Gaussian or box filters. It was experimentally demonstrated that such a simple post-processing of predictions from simple blending of two top visual models improves the F1-score of facial expression recognition up to 7%. At the same time, the mean Concordance Correlation Coefficient (CCC) of valence and arousal is increased by up to 1.25 times compared to each model's frame-level predictions. As a result, our final performance score on the validation set from the multi-task learning challenge is 4.5 times higher than the baseline (1.494 vs 0.32).

show abstract

ABAW: Learning from Synthetic Data & Multi-task Learning Challenges

Kollias

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Affective Behaviour Analysis Using Pretrained Model with Facial Prior

Cited by 7 publications

References 18 publications

Exploring Expression-related Self-supervised Learning for Affective Behaviour Analysis

Exploring Expression-related Self-supervised Learning for Affective Behaviour Analysis

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

ABAW: Learning from Synthetic Data & Multi-task Learning Challenges

Contact Info

Product

Resources

About