Cross-Modal Representation Learning for Lightweight and Accurate Facial Action Unit Detection

Chen, Ying‐Jie; Wu, Han; Wang, Tao; Wang, Yizhou; Liang, Yun

doi:10.1109/lra.2021.3098944

Cited by 7 publications

(4 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since then, AUs have become a central component of many applications, including human activity recognition and behavior understanding [15], facial expression recognition (FER) [16], video games [17], car driver attention monitoring systems [18] and remote health monitoring [19]. AUs are combinations of facial muscle movements and are the basic components of facial expressions [20]. The development of AU detection systems has been a longstanding challenge in artificial intelligence, with early approaches relying on classical methods such as Gabor filters, principal component analysis (PCA) [21], Support Vector Machines (SVM) [3], and k-Nearest Neighbor classifiers (KNN) [22].…”

Section: A Background and Related Workmentioning

confidence: 99%

“…Moreover, to remind that the main objective of this work is to propose a lightweight model with the minimum number of parameters for AU detection in resource-constrained systems. Zhang [5] VGGNet >138 FSNet [20] ResNet-50(customized) 8,19 ARL [25] VGGNet >138 STRAL [30] VGGNet >138 LGRNet [78] BiLSTM >4 MCFE [79] DENSnet-121 >3 CWCF [80] ResNet-9 >26 IDENnet [81] LightCNN >6,572 JAU [82] Resnet-18 >11 This work Attention-CNN 1,5…”

Section: B Model Sizementioning

confidence: 99%

“…To assess the effectiveness of our proposed model in comparison to the state-of-the-art methods, we evaluated our method against various existing models on the CK+, DISFA, and BP4D datasets. The models considered for comparison include Zhang [5], LibreFace [65], FS [20], STRAL [30], IDEN [81], LGR [78], MCFE [79], LUO [83], CWCF [80], JAU [82] for BP4D and DISFA datasets, and JPML [84], Simge [85], Chen1 [41], Chen2 [41], Elef [86] for the CK+ dataset.…”

Section: A Au Detection Comparison On Ck+ Disfa and Bp4d Datasetsmentioning

confidence: 99%

See 2 more Smart Citations

Attention-Enabled Lightweight Neural Network Architecture for Detection of Action Unit Activation

Deramgozin,

Jovanovic,

Arevalillo-Herráez

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Facial Action Unit (AU) detection is of major importance in a broad range of artificial intelligence applications such as healthcare, Facial Expression Recognition (FER), and mental state analysis. In this paper, we present an innovative, resource-efficient facial AU detection model, embedding both spatial and channel attention mechanisms into a convolutional neural network (CNN). Along with a unique data input system leveraging image data and binary-encoded AU activation labels, our model enhances AU detection capabilities while simultaneously offering interpretability for FER systems. In contrast to existing state-of-the-art models, our proposal's streamlined architecture, combined with superior performance, establishes it as an ideal solution for resource-limited environments like mobile and embedded systems with computational constraints. The model was trained and evaluated utilizing the BP4D, CK+, DISFA, FER2013+, and RAF-DB datasets, with the latter two being particularly significant as they represent wild datasets for facial expression recognition. These datasets encompass ground truth emotions matched with corresponding AU activations according to the Facial Action Coding System. Various metrics, including F1 score, accuracy, and Euclidean distance, showcase its effectiveness in AU detection and interpretability.

show abstract

Section: A Background and Related Workmentioning

confidence: 99%

Section: B Model Sizementioning

confidence: 99%

Section: A Au Detection Comparison On Ck+ Disfa and Bp4d Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Attention-Enabled Lightweight Neural Network Architecture for Detection of Action Unit Activation

Deramgozin,

Jovanovic,

Arevalillo-Herráez

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…According to Facial Action Coding System (FACS) (Friesen and Ekman 1978), facial action units (AUs), defined as the combinations of facial muscle movements, can describe almost all facial behaviors, which is essential for fine-grained facial behavior analysis. In recent years, deep learning has proved its efficacy and efficiency in facial action unit recognition task (Cui et al 2020;Chen et al 2021b;Yang et al 2021;Song et al 2021b), but there is still room for improvement since some inherent nature of AU has not been fully exploited.…”

Section: Introductionmentioning

confidence: 99%

Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition

Chen

Wang

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Subject-invariant facial action unit (AU) recognition remains challenging for the reason that the data distribution varies among subjects. In this paper, we propose a causal inference framework for subject-invariant facial action unit recognition. To illustrate the causal effect existing in AU recognition task, we formulate the causalities among facial images, subjects, latent AU semantic relations, and estimated AU occurrence probabilities via a structural causal model. By constructing such a causal diagram, we clarify the causal-effect among variables and propose a plug-in causal intervention module, CIS, to deconfound the confounder Subject in the causal diagram. Extensive experiments conducted on two commonly used AU benchmark datasets, BP4D and DISFA, show the effectiveness of our CIS, and the model with CIS inserted, CISNet, has achieved state-of-the-art performance.

show abstract

AUPro: Multi-label Facial Action Unit Proposal Generation for Sequence-Level Analysis

Chen

Zhang

Chen

et al. 2021

Neural Information Processing

View full text Add to dashboard Cite

Cross-Modal Representation Learning for Lightweight and Accurate Facial Action Unit Detection

Cited by 7 publications

References 42 publications

Attention-Enabled Lightweight Neural Network Architecture for Detection of Action Unit Activation

Attention-Enabled Lightweight Neural Network Architecture for Detection of Action Unit Activation

Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition

AUPro: Multi-label Facial Action Unit Proposal Generation for Sequence-Level Analysis

Contact Info

Product

Resources

About