Affective Behavior Analysis Using Action Unit Relation Graph and Multi-task Cross Attention

Nguyen, Dang-Khanh; Pant, Sudarshan; Ho, Ngoc-Huynh; Lee, Guee-Sang; Kim, Soo-Hyung; Yang, Hyung-Jeong

doi:10.1007/978-3-031-25075-0_10

Cited by 4 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As one can see, smoothing does not work for AU detection but can significantly improve the results for other tasks: up to 0.06 difference in F1-score for EXPR classification and up to 0.06 difference in mean CCC for VA prediction. Moreover, the smoothing works nicely even for blending the best ---0.981 SMMEmotionNet [23] 0.3648 0.2617 0.4737 1.1002 Two-Aspect Information Interaction [31] 0.515 0.207 0.385 1.107 SS-MFAR [4] 0.397 0.235 0.493 1.125 EfficientNet-B2 [27] 0.384 0.302 0.461 1.147 MAE+ViT [20] 0.4588 0.3028 0.5054 1.2671 Cross-attentive module [23] 0.499 0.333 0.456 1.288 MT-EmotiEffNet + OpenFace [29] 0.447 0.357 0.496 1.300 MAE+Transformer [37] 0 frame-level 0.4847 0.3578 0.5194 1.3619 + MT-EmotiDDAMFN smoothing, tAU = 0.5 0.5578 0.4168 0.5194 1.4939 models (Fig. 5).…”

Section: Multi-task Learning Challengementioning

confidence: 97%

“…At the same time, the hybrid CNN (Convolutional Neural Network)-Transformer [33] with a fusion of ResNet-18 and a spatial transformer took the 5th place. Slightly better results have been obtained by the cross-attentive module and a facial graph that captures the association among action units [23]. The EfficientNet model pre-trained in Multi-Task setting (MT-EmotiEffNet) took the third place [29].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Savchenko¹

2021

2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY)

120

View full text Add to dashboard Cite

In this paper, we describe the results of the HSEmotion team in two tasks of the seventh Affective Behavior Analysis in-thewild (ABAW) competition, namely, multi-task learning for simultaneous prediction of facial expression, valence, arousal, and detection of action units, and compound expression recognition. We propose an efficient pipeline based on frame-level facial feature extractors pre-trained in multi-task settings to estimate valence-arousal and basic facial expressions given a facial photo. We ensure the privacy-awareness of our techniques by using the lightweight architectures of neural networks, such as MT-EmotiDDAMFN, MT-EmotiEffNet, and MT-EmotiMobileFaceNet, that can run even on a mobile device without the need to send facial video to a remote server. It was demonstrated that a significant step in improving the overall accuracy is the smoothing of neural network output scores using Gaussian or box filters. It was experimentally demonstrated that such a simple post-processing of predictions from simple blending of two top visual models improves the F1-score of facial expression recognition up to 7%. At the same time, the mean Concordance Correlation Coefficient (CCC) of valence and arousal is increased by up to 1.25 times compared to each model's frame-level predictions. As a result, our final performance score on the validation set from the multi-task learning challenge is 4.5 times higher than the baseline (1.494 vs 0.32).

show abstract

Section: Multi-task Learning Challengementioning

confidence: 97%

Section: Related Workmentioning

confidence: 99%

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Savchenko¹

2021

2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY)

120

View full text Add to dashboard Cite

show abstract

“…Song et al [47] construct a co-occurrence knowledge graph and a spatio-temporal Transformer module to capture the temporal and spatial relations of AUs. Nguyen et al [36] use a facial graph to capture the association among action units for the multi-task learning challenge and they ranked 4th in multi-task challenge at the ABAW competition 2022. These works exhibit the effectiveness of modeling AU relationships.…”

Section: Graph-based Au Recognition Approachesmentioning

confidence: 99%

“…Some of the AU detection approaches in the previous ABAW Competitions [16,17,24] fuse multimodal features including video and audio to provide multidimensional information to predict AUs' occurrence [13,14,50,58]. Meanwhile, other studies found that AU detection performance can be benefited from multi-task learning [3,13,36,56], i.e., jointly conducting expression recognition or valence/arousal estimation provides helpful cues for AU detection. Moreover, temporal models such as GRU [6] or Transformer [48] are also introduced to model temporal dynamics among consecutive frames [37,50].…”

Section: Introductionmentioning

confidence: 99%

Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detection

Wang¹,

Song²,

Luo³

et al. 2023

Preprint

View full text Add to dashboard Cite

This paper presents our Facial Action Units (AUs) recognition submission to the fifth Affective Behavior Analysis in-the-wild Competition (ABAW). Our approach consists of three main modules: (i) a pre-trained facial representation encoder which produce a strong facial representation from each input face image in the input sequence; (ii) an AUspecific feature generator that specifically learns a set of AU features from each facial representation; and (iii) a spatiotemporal graph learning module that constructs a spatiotemporal graph representation. This graph representation describes AUs contained in all frames and predicts the occurrence of each AU based on both the modeled spatial information within the corresponding face and the learned temporal dynamics among frames. The experimental results show that our approach outperformed the baseline and the spatio-temporal graph representation learning allows our model to generate the best results among all ablated systems. Our model ranks at the 4th place in the AU recognition track at the 5th ABAW Competition.

show abstract

“…The in-the-wild affective behavior analysis is a study to understand individuals' emotions and moods through their facial expressions, action behavior and physical characteristics. It has been an important research direction in the fields of mental health treatment, human-computer interaction, and marketing research [1,5,20,24]. Usually, three main representations, i.e., Action Units (AU), Valence-Arousal * Corresponding author (VA), and the basic facial expressions(e.g., happy, sad, and neutral) are used to gain insight into how individuals express and experience emotions.…”

Section: Introductionmentioning

confidence: 99%

Spatial-temporal Transformer for Affective Behavior Analysis

Zou¹,

Wang²,

Wen³

et al. 2023

Preprint

View full text Add to dashboard Cite

The in-the-wild affective behavior analysis has been an important study. In this paper, we submit our solutions for the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW), which includes V-A Estimation, Facial Expression Classification and AU Detection Sub-challenges. We propose a Transformer Encoder with Multi-Head Attention framework to learn the distribution of both the spatial and temporal features. Besides, there are virious effective data augmentation strategies employed to alleviate the problems of sample imbalance during model training. The results fully demonstrate the effectiveness of our proposed model based on the Aff-Wild2 dataset.

show abstract

Affective Behavior Analysis Using Action Unit Relation Graph and Multi-task Cross Attention

Cited by 4 publications

References 13 publications

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detection

Spatial-temporal Transformer for Affective Behavior Analysis

Contact Info

Product

Resources

About