High Fidelity Face Manipulation with Extreme Poses and Expressions

Abstract

Face manipulation has shown remarkable advances with the flourish of Generative Adversarial Networks. However, due to the difficulties of controlling the structure and texture in high-resolution, it is challenging to simultaneously model pose and expression during manipulation. In this paper, we propose a novel framework that simplifies face manipulation with extreme pose and expression into two correlated stages: a boundary prediction stage and a disentangled face synthesis stage. In the first stage, we propose to use a boundary image for joint pose and expression modeling. An encoder-decoder network is employed to predict the boundary image of the target face in a semi-supervised way. Pose and expression estimators are employed to improve the prediction accuracy. In the second stage, the predicted boundary image and the original face are encoded into the structure and texture latent space by two encoder networks respectively. A proxy network and a feature threshold loss are further imposed to disentangle the latent space. Furthermore, considering the lack of high-resolution face databases to verify the effectiveness of our method, we collect a new high quality Multi-View Face (MVF-HQ) database in 6000 × 4000 resolution. It contains 120,283 images from 479 identities with diverse pose, expression and illumination variants, which is much larger in scale and much higher in resolution than the current public high-resolution face manipulation databases. We expect it to push forward the advance of face manipulation. Qualitative and quantitative experiments on four databases show that our method dramatically improves the visualization of face manipulation.

LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements

“…Recently, the use of VAEs [11] and adversarial image-to-image translation networks have become extremely popular for editing facial expressions [44,106,24], with or without paired data for training. Some of these models use attention masks [73], facial shape information [33,31] or exemplar videos [83] to guide the model in this task.…”

Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars

“…Thus, several methods [26,36,39,41] use generators based on direct synthesis, where the image is generated using a sequence of convolutional operators, interleaved with elementwise non-linearities, and normalizations. Person identity information may be injected into such architecture, either with a lengthy learning process (in the many-shot scenario) [26,39] or by using adaptive normalizations conditioned on person embeddings [13,36,41]. The method [41] effectively combines both approaches by injecting identity through adaptive normalizations, and then fine-tuning the resulting generator on the fewshot learning set.…”

“…The first works required a video [26,39] or even multiple videos [28,35] to create a neural network that can synthesize talking head view of a person. Most recently, several works [13,17,33,33,36,37,41] presented systems that Existing few-shot neural head avatar systems achieve remarkable results. Yet, unlike some of the graphics-based avatars, the neural systems are too slow to be deployed on mobile devices and require a high-end desktop GPU to run in real-time.…”

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition

“…Recently, the great progress of high-quality face synthesis [38,5,33,39] has made "recognition via generation" possible. TP-GAN [16] and CAPG-GAN [13] introduce face synthesis to improve the quantitative performance of large pose face recognition.…”