Face Alignment in Full Pose Range: A 3D Total Solution

Zhu, Xiangyu; Liu, Xiaoming; Lei, Zhen; Li, Stan Z.

doi:10.1109/tpami.2017.2778152

Cited by 345 publications

(333 citation statements)

References 66 publications

Supporting

Mentioning

333

Contrasting

Order By: Relevance

“…However we limit our training to faces that are nearly frontal. To do this we use pose estimation software [18] based on the model proposed in [46] to select faces whose roll, pitch and yaw angles are smaller 10°.…”

Section: Datasetsmentioning

confidence: 99%

Realistic Speech-Driven Facial Animation with GANs

2019

View full text Add to dashboard Cite

Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.

show abstract

Section: Datasetsmentioning

confidence: 99%

Realistic Speech-Driven Facial Animation with GANs

2019

View full text Add to dashboard Cite

show abstract

“…Fitting 3DMM into 2D images is a difficult optimization problem, an extensive discussion of which falls beyond the scope of this work. In our case, we use the state-of-the-art 3DDFA method proposed in [18,23], which trains a fully convolutional network to regress the parameters p from an input 2D image in a cascaded manner. In order to provide a better initialization for the 3DMM fitting, we detect 68 facial landmarks for every frame using FAN [24], which is also used to accurately crop the input image.…”

Section: Pose Augmentationmentioning

confidence: 99%

“…Our contributions can be summarised as follows: (i) We describe a method to construct a large-pose synthetic database for lip-reading. Our method capitalizes on robust 3DMM fitting [18], which allows us to take as input a frontal facial image and render it in any arbitrary pose. Using this method, we derive a database that extends the large-scale but mostly frontal database of LRW, which we call LRW in Large Poses (LP).…”

Section: Introductionmentioning

confidence: 99%

Towards Pose-Invariant Lip-Reading

Cheng

Tzimiropoulos

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Lip-reading models have been significantly improved recently thanks to powerful deep learning architectures. However, most works focused on frontal or near frontal views of the mouth. As a consequence, lip-reading performance seriously deteriorates in non-frontal mouth views. In this work, we present a framework for training pose-invariant lip-reading models on synthetic data instead of collecting and annotating non-frontal data which is costly and tedious. The proposed model significantly outperforms previous approaches on nonfrontal views while retaining the superior performance on frontal and near frontal mouth views. Specifically, we propose to use a 3D Morphable Model (3DMM) to augment LRW, an existing large-scale but mostly frontal dataset, by generating synthetic facial data in arbitrary poses. The newly derived dataset, is used to train a state-of-the-art neural network for lip-reading. We conducted a cross-database experiment for isolated word recognition on the LRS2 dataset, and reported an absolute improvement of 2.55%. The benefit of the proposed approach becomes clearer in extreme poses where an absolute improvement of up to 20.64% over the baseline is achieved.

show abstract

“…The AFLW2000-3D dataset consists of fitted 3D faces and large-pose images for the first 2000 images of the AFLW database [12]. As it was done in [31], we evaluate the capacities of our method to deal with non-frontal poses by training on 300W-LP and testing on AFLW2000-3D. [31], we report accuracy for each pose range separately, as well as the mean across those three pose ranges.…”

Section: D Face Alignmentmentioning

confidence: 99%

“…As it was done in [31], we evaluate the capacities of our method to deal with non-frontal poses by training on 300W-LP and testing on AFLW2000-3D. [31], we report accuracy for each pose range separately, as well as the mean across those three pose ranges.…”

Section: D Face Alignmentmentioning

confidence: 99%

Tree-Gated Deep Mixture-of-Experts for Pose-Robust Face Alignment

Arnaud

Dapogny

Bailly

2020

IEEE Trans. Biom. Behav. Identity Sci.

View full text Add to dashboard Cite

Face alignment consists of aligning a shape model on a face image. It is an active domain in computer vision as it is a preprocessing for a number of face analysis and synthesis applications. Current state-of-the-art methods already perform well on "easy" datasets, with moderate head pose variations, but may not be robust for "in-the-wild" data with poses up to 90 • . In order to increase robustness to an ensemble of factors of variations (e.g. head pose or occlusions), a given layer (e.g. a regressor or an upstream CNN layer) can be replaced by a Mixture of Experts (MoE) layer that uses an ensemble of experts instead of a single one. The weights of this mixture can be learned as gating functions to jointly learn the experts and the corresponding weights. In this paper, we propose to use tree-structured gates which allows a hierarchical weighting of the experts (Tree-MoE). We investigate the use of Tree-MoE layers in different contexts in the frame of face alignment with cascaded regression, firstly for emphasizing relevant, more specialized feature extractors depending of a high-level semantic information such as head pose (Pose-Tree-MoE), and secondly as an overall more robust regression layer. We perform extensive experiments on several challenging face alignment datasets, demonstrating that our approach outperforms the state-of-the-art methods.

show abstract

Face Alignment in Full Pose Range: A 3D Total Solution

Cited by 345 publications

References 66 publications

Realistic Speech-Driven Facial Animation with GANs

Realistic Speech-Driven Facial Animation with GANs

Towards Pose-Invariant Lip-Reading

Tree-Gated Deep Mixture-of-Experts for Pose-Robust Face Alignment

Contact Info

Product

Resources

About