Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition

Li, Guanbin; Zhu, Xin; Zeng, Yirui; Wang, Qing; Li, Lin

doi:10.1609/aaai.v33i01.33018594

Cited by 130 publications

(111 citation statements)

References 25 publications

Supporting

Mentioning

109

Contrasting

Order By: Relevance

“…There are many works about AU occurrence detection [8,22,23,29], which is considered as a multilabel classification problem. For instance, several works considered the relationships on various AUs and modeled AU interrelations to improve recognition accuracy [4,21,26]. However, most works relied on probabilistic graphical models with manually extracted features [5,53], which limits the extension for deep learning.…”

Section: Facial Au Detectionmentioning

confidence: 99%

AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition

Xie

Shuai

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

127

111

View full text Add to dashboard Cite

Micro-expressions (MEs) are important clues for reflecting the real feelings of humans, and micro-expression recognition (MER) can thus be applied in various real-world applications. However, it is difficult to perceive and interpret MEs correctly. With the advance of deep learning technologies, the accuracy of micro-expression recognition is improved but still limited by the lack of large-scale datasets. In this paper, we propose a novel micro-expression recognition approach by combining Action Units (AUs) and emotion category labels. Specifically, based on facial muscle movements, we model different AUs based on relational information and integrate the AUs recognition task with MER. Besides, to overcome the shortcomings of limited and imbalanced training samples, we propose a data augmentation method that can generate nearly indistinguishable image sequences with AU intensity of real-world micro-expression images, which effectively improve the performance and are compatible with other micro-expression recognition methods. Experimental results on three mainstream micro-expression datasets, i.e., CASME II, SAMM, and SMIC, manifest that our approach outperforms other state-of-the-art methods on both single database and cross-database micro-expression recognition. CCS CONCEPTS • Computing methodologies → Neural networks; Computer vision; Image representations.

show abstract

Section: Facial Au Detectionmentioning

confidence: 99%

AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition

Xie

Shuai

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

127

111

View full text Add to dashboard Cite

show abstract

“…Comparison with single-modal based methods . To prove that multimodal provides additional valuable information for AU detection, we first compare our method to the single modality based methods, including Deep Structure Inference Network (DSIN) [7], Joint AU Detection and Face Alignment (JAA) [31], Optical Flow network (OF-Net) [42], Local relationship learning with Personspecific shape regularization (LP-Net) [29], Semantic Relationships Embedded Representation Learning ( SRERL) [20], and ResNet18.…”

Section: 31mentioning

confidence: 99%

Adaptive Multimodal Fusion for Facial Action Units Recognition

Yang

Wang

Liu

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Multimodal facial action units (AU) recognition aims to build models that are capable of processing, correlating, and integrating information from multiple modalities (i.e., 2D images from a visual sensor, 3D geometry from 3D imaging, and thermal images from an infrared sensor). Although the multimodel data can provide rich information, there are two challenges that have to be addressed when learning from multimodal data: 1) the model must capture the complex cross-modal interactions in order to utilize the additional and mutual information effectively; 2) the model must be robust enough in the circumstance of unexpected data corruptions during testing, in case of a certain modality missing or being noisy. In this paper, we propose a novel Adaptive Multimodal Fusion method (AMF) for AU detection, which learns to select the most relevant feature representations from different modalities by a re-sampling procedure conditioned on a feature scoring module. The feature scoring module is designed to allow for evaluating the quality of features learned from multiple modalities. As a result, AMF is able to adaptively select more discriminative features, thus increasing the robustness to missing or corrupted modalities. In addition, to alleviate the over-fitting problem and make the model generalize better on the testing data, a cut-switch multimodal data augmentation method is designed, by which a random block is cut and switched across multiple modalities. We have conducted a thorough investigation on two public multimodal AU datasets, BP4D and BP4D+, and the results demonstrate the effectiveness of the proposed method. Ablation studies on various circumstances also show that our method remains robust to missing or noisy modalities during tests. CCS CONCEPTS • Computing methodologies → Activity recognition and understanding; Biometrics; Image representations.

show abstract

“…Recently, Li et al [10] proposed the AU semantic relationship embedded representation learning (SRERL) framework to combine facial AU detection and Gated Graph Neural Network (GGNN) [12] and achieved good results. But the commonly used Graph Convolutional Network (GCN) for classification task with relation modeling is adopted for AU relation modeling in our proposed method, while the Gated Graph Neural Network (GGNN) adopted in [13] is inspired by GRU and mainly used for the task of Visual Question Answering and Semantic Segmentation.…”

Section: Related Workmentioning

confidence: 99%

Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection

Liu

Dong

Zhang

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Most existing AU detection works considering AU relationships are relying on probabilistic graphical models with manually extracted features. This paper proposes an end-to-end deep learning framework for facial AU detection with graph convolutional network (GCN) for AU relation modeling, which has not been explored before. In particular, AU related regions are extracted firstly, latent representations full of AU information are learned through an auto-encoder. Moreover, each latent representation vector is feed into GCN as a node, the connection mode of GCN is determined based on the relationships of AUs. Finally, the assembled features updated through GCN are concatenated for AU detection. Extensive experiments on BP4D and DISFA benchmarks demonstrate that our framework significantly outperforms the state-of-the-art methods for facial AU detection. The proposed framework is also validated through a series of ablation studies.

show abstract

Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition

Cited by 130 publications

References 25 publications

AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition

AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition

Adaptive Multimodal Fusion for Facial Action Units Recognition

Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection

Contact Info

Product

Resources

About