Conditional adversarial synthesis of 3D facial action units

Abstract

Employing deep learning-based approaches for finegrained facial expression analysis, such as those involving the estimation of Action Unit (AU) intensities, is difficult due to the lack of a large-scale dataset of real faces with sufficiently diverse AU labels for training. In this paper, we consider how AU-level facial image synthesis can be used to substantially augment such a dataset. We propose an AU synthesis framework that combines the well-known 3D Morphable Model (3DMM), which intrinsically disentangles expression parameters from other face attributes, with models that adversarially generate 3DMM expression parameters conditioned on given target AU labels, in contrast to the more conventional approach of generating facial images directly. In this way, we are able to synthesize new combinations of expression parameters and facial images from desired AU labels. Extensive quantitative and qualitative results on the benchmark DISFA dataset demonstrate the effectiveness of our method on 3DMM facial expression parameter synthesis and data augmentation for deep learning-based AU intensity estimation.

Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment

“…(6) is to alleviate the data imbalance problem. For most facial AU detection benchmarks, the occurrence rates of AUs are imbalanced [13,12]. Since AUs are not mutually independent, imbalanced training data has a bad influence on this multi-label learning task.…”

Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection

“…Recently, Li et al [10] proposed the AU semantic relationship embedded representation learning (SRERL) framework to combine facial AU detection and Gated Graph Neural Network (GGNN) [12] and achieved good results. But the commonly used Graph Convolutional Network (GCN) for classification task with relation modeling is adopted for AU relation modeling in our proposed method, while the Gated Graph Neural Network (GGNN) adopted in [13] is inspired by GRU and mainly used for the task of Visual Question Answering and Semantic Segmentation. In addition, our method has only about 2.3 million parameters, but SRERL [13] has more than 138 million parameters.…”

“…The trade-off weight w i is introduced to alleviate the data imbalance problem. For most facial AU detection benchmarks, the occurrence rates of AUs are imbalanced [13,14]. Since AUs are not mutually independent, imbalanced training data has a bad influence on this multi-label learning task.…”

Region Based Adversarial Synthesis of Facial Action Units

“…However, the resolution of its generated facial image is only 32 × 32, and the generated facial images with AU labels are not well quantitatively evaluated. By integrating with the 3D Morphable Model (3DMM), Liu et al [13] proposed an 3D AU synthesis framework to transfer AUs in the range of intensities, however, some texture details are lost for certain AUs because of the limitations of 3D face model. In addition, while generating a single AU, all these above mentioned methods can damage the other AU without keeping the identity information untouched.…”

“…It is difficult for the method to synthesize a single AU respectively without keeping the other AU untouched. Liu et al [13] uses 3D Morphable Model (3DMM) to achieve AU synthesis, where the transformation from a image to 3D model damages the texture details of the original images.…”