Beyond 3DMM Space: Towards Fine-Grained 3D Face Reconstruction

Zhu, Xiangyu; Yang, Fan; Huang, Di; Yu, Changqiu; Wang, Hao; Guo, Jianzhu; Lei, Zhen; Li, Stan Z.

doi:10.1007/978-3-030-58598-3_21

Cited by 29 publications

(13 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…detailed shape [46], [47], [48], [49], [50], [51], [52], [53]. Nonparametric methods directly predict the facial shape in the form of mesh vertices [47], [54], [55], [56], [57], [58], depth [59], or volume [60]. Comparing to parametric model, nonparametric representation has higher degrees of freedom and therefore can express finer shape details.…”

Section: Related Workmentioning

confidence: 99%

FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction

Zhu¹,

Yang²,

Guo³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we present a large-scale detailed 3D face dataset, FaceScape, and the corresponding benchmark to evaluate single-view facial 3D reconstruction. By training on FaceScape data, a novel algorithm is proposed to predict elaborate riggable 3D face models from a single image input. FaceScape dataset provides 18,760 textured 3D faces, captured from 938 subjects and each with 20 specific expressions. The 3D models contain the pore-level facial geometry that is also processed to be topologically uniformed. These fine 3D facial models can be represented as a 3D morphable model for rough shapes and displacement maps for detailed geometry. Taking advantage of the large-scale and high-accuracy dataset, a novel algorithm is further proposed to learn the expression-specific dynamic details using a deep neural network. The learned relationship serves as the foundation of our 3D face prediction system from a single image input. Different than the previous methods, our predicted 3D models are riggable with highly detailed geometry under different expressions. We also use FaceScape data to generate the in-the-wild and in-the-lab benchmark to evaluate recent methods of single-view face reconstruction. The accuracy is reported and analyzed on the dimensions of camera pose and focal length, which provides a faithful and comprehensive evaluation and reveals new challenges. The unprecedented dataset, benchmark, and code have been released to the public for research purpose 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction

Zhu¹,

Yang²,

Guo³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…With the development of deep learning, some methods use CNN to estimate the 3D Morphable Model parameters [12,30]. Using 3DMM parameters, we can obtain a series of 3D representations, such as 3D face shape, texture, and UV texture map [8,31]. In addition, neural renderer [9,15,22] can be used to fit 3D face models to 2D images.…”

Section: D Face Reconstructionmentioning

confidence: 99%

SOGAN: 3D-Aware Shadow and Occlusion Robust GAN for Makeup Transfer

Lyu,

Dong,

Peng

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, virtual makeup applications have become more and more popular. However, it is still challenging to propose a robust makeup transfer method in the real-world environment. Current makeup transfer methods mostly work well on good-conditioned clean makeup images, but transferring makeup that exhibits shadow and occlusion is not satisfying. To alleviate it, we propose a novel makeup transfer method, called 3D-Aware Shadow and Occlusion Robust GAN (SOGAN). Given the source and the reference faces, we first fit a 3D face model and then disentangle the faces into shape and texture. In the texture branch, we map the texture to the UV space and design a UV texture generator to transfer the makeup. Since human faces are symmetrical in the UV space, we can conveniently remove the undesired shadow and occlusion from the reference image by carefully designing a Flip Attention Module (FAM). After obtaining cleaner makeup features from the reference image, a Makeup Transfer Module (MTM) is introduced to perform accurate makeup transfer. The qualitative and quantitative experiments demonstrate that our SOGAN not only achieves superior results in shadow and occlusion situations but also performs well in large pose and expression variations.

show abstract

“…Parametric Method: With 3DMM [4] proposed, 3D face modeling can be formulated in a procedure of parametric optimization [38,37,73]. Recently, deep neural networks are introduced to regress 3DMM parameter from input image [71,35,9,12,72] by learning from generated ground truth. With neural rendering approach such as [22], methods are proposed to leverage image reconstruction loss to train the model in weakly or un-supervised manner [49,36,14], or improve 3DMM with more nonlinear feasibility [51,50,48,27,6].…”

Section: Related Workmentioning

confidence: 99%

“…Meanwhile, as these single-view guided methods may suffer from 2D ambiguity, other 3DMM-based works are proposed to leverage multiview consistency [53,47,62,57,2]. While 3DMM provides reliable priors for 3D face modeling, it also brings potential drawbacks: as built from a small amount of subjects (e.g., BFM [33] with 200 subjects) and rigidly controlled conditions, models may be fragile to large variations of identity [72], and have limitations on building teeth, skin details or anatomic grounded muscles [10].…”

Section: Introductionmentioning

confidence: 99%

“…LAP first aggregates consistent face factors of an identity from in-the-wild photo collection, and then personalizes such factors to reconstruct a scene-specific face for a target image of the same ID. Con-MI-Consistency Shape Assumption Supervision [71,35,9,12,72] × 3DMM 3DMM paramter [49,36,14,51,50,48,27 ii) With a novel relaxed curriculum aggregation method, LAP is able to predict ID-consistent face factors against large facial variations of in-the-wild photo set.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection

Zhang

Chen

et al. 2021

Preprint

View full text Add to dashboard Cite

Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. While plausible facial details are predicted, the models tend to over-depend on local color appearance and suffer from ambiguous noise. To address such problem, this paper presents a novel Learning to Aggregate and Personalize (LAP) framework for unsupervised robust 3D face modeling. Instead of using controlled environment, the proposed method implicitly disentangles ID-consistent and scenespecific face from unconstrained photo set. Specifically, to learn ID-consistent face, LAP adaptively aggregates intrinsic face factors of an identity based on a novel curriculum learning approach with relaxed consistency loss. To adapt the face for a personalized scene, we propose a novel attribute-refining network to modify ID-consistent face with target attribute and details. Based on the proposed method, we make unsupervised 3D face modeling benefit from meaningful image facial structure and possibly higher resolutions. Extensive experiments on benchmarks show LAP recovers superior or competitive face shape and texture, compared with state-of-the-art (SOTA) methods with or without prior and supervision 1 .

show abstract

Beyond 3DMM Space: Towards Fine-Grained 3D Face Reconstruction

Cited by 29 publications

References 41 publications

FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction

FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction

SOGAN: 3D-Aware Shadow and Occlusion Robust GAN for Makeup Transfer

Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection

Contact Info

Product

Resources

About