Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set

Deng, Yu; Yang, Jiaolong; Xu, Sicheng; Chen, Dong; Jia, Yingmin; Tong, Xin

doi:10.1109/cvprw.2019.00038

Cited by 643 publications

(652 citation statements)

References 60 publications

Supporting

Mentioning

644

Contrasting

Unclassified

Order By: Relevance

“…The other two results were generated using pre-trained deeplearning models. While the method from [49] estimates the shape of a face by inferring the shape parameters of a 3DMM, a detailed facial shape with a UV displacement map is generated in [36]. Although our method is based on iterative linear optimization, the quality of the recovered 3D shapes is comparable to that of the latest methods based on deeplearning.…”

Section: B 3d Shape Recoverymentioning

confidence: 98%

See 1 more Smart Citation

Local Spherical Harmonics for Facial Shape and Albedo Estimation

et al. 2020

View full text Add to dashboard Cite

In this paper, we present a novel facial albedo and 3D shape recovery method with a local spherical harmonic illumination model. From a face in an image, the proposed method can produce a highquality 3D shape and albedo using a novel parameterization of local illuminations. Because a facial shape is partially convex, a single spherical harmonics is generally used for the illumination of a face within a constrained illumination environment. However, when a facial image is captured in an unconstrained scene, the illumination is inappropriately estimated due to the presence of shadow and specular reflections. To address this issue, we propose a novel local spherical harmonic illumination model for representing the illumination of a face. Unlike the existing parameterization of local illumination, our local spherical harmonic illumination model utilizes a smooth weight function for the seamless representation of natural illumination. Therefore, the albedo and shape information in an image can be precisely estimated using the first-order spherical harmonics. For accurate estimation of albedo, we also utilize facial albedo statistics to prevent the estimated albedo from becoming biased toward input image. Furthermore, we developed an accurate and reliable 3D shape reconstruction method from a normal map based on tetrahedron-based deformation. Comparing to the Laplacian deformation based method, our method is applicable to any mesh regardless of its structure. Through rigorous experiments, we demonstrate that the proposed local spherical harmonic illumination model is effective in estimating the complex illumination and can recover a highquality facial albedo and 3D shape.

show abstract

Section: B 3d Shape Recoverymentioning

confidence: 98%

“…Comparison of results for 3D facial shape recovery from the Bosphorus database. We compare our proposed method to the methods proposed by Jiang et al [17], Deng et al [49], and Chen et al [36]. Note that ground truth was constructed by meshing the noisy depth map provided in the database.…”

Section: Rs-mse S-ssimmentioning

confidence: 99%

Local Spherical Harmonics for Facial Shape and Albedo Estimation

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Feng et al [16] designed a 2D representation called UV position map to record 3D positions of a complete human face. Deng et al [14] directly regressed a group of parameters based on 3DMM [6,7,31]. All these works can reconstruct the 3D face model from a single image but cannot recover geometry details.…”

Section: Related Workmentioning

confidence: 99%

“…Although some of these methods are capable of reconstructing high-quality 3D face models with both low-frequency structures and high-frequency details like wrinkles and pores, the hardware environment is hard to set up and the underlying optimization problem is not easy to solve. For this reason, 3D face reconstruction from a single image has attracted wide attention, with many works focusing on reconstruction from an "in-the-wild" im-age [35,17,14,18]. Although most of them can reconstruct accurate low-frequency facial structures, few can recover fine facial details.…”

Section: Introductionmentioning

confidence: 99%

Lightweight Photometric Stereo for Facial Details Recovery

Guo

Deng

Zhang

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Recently, 3D face reconstruction from a single image has achieved great success with the help of deep learning and shape prior knowledge, but they often fail to produce accurate geometry details. On the other hand, photometric stereo methods can recover reliable geometry details, but require dense inputs and need to solve a complex optimization problem. In this paper, we present a lightweight strategy that only requires sparse inputs or even a single image to recover high-fidelity face shapes with images captured under near-field lights. To this end, we construct a dataset containing 84 different subjects with 29 expressions under 3 different lights. Data augmentation is applied to enrich the data in terms of diversity in identity, lighting, expression, etc. With this constructed dataset, we propose a novel neural network specially designed for photometric stereo based 3D face reconstruction. Extensive experiments and comparisons demonstrate that our method can generate high-quality reconstruction results with one to three facial images captured under near-field lights. Our full framework is available at https://github.com/Juyong/FacePSNet.

show abstract

“…One line of CNN-based 3D face reconstruction work offers the promise of overcoming this reliance using model-based autoencoders for self-supervision [29,13,10,28,6]. Here, a 3DMM and a differentiable renderer are used as a model-based decoder such that a trainable encoder (a CNN) can learn to regress semantically meaningful model parameters.…”

Section: Introductionmentioning

confidence: 99%

“Look Ma, No Landmarks!” – Unsupervised, Model-Based Dense Face Alignment

Koizumi

Smith

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In this paper, we show how to train an image-to-image network to predict dense correspondence between a face image and a 3D morphable model using only the model for supervision. We show that both geometric parameters (shape, pose and camera intrinsics) and photometric parameters (texture and lighting) can be inferred directly from the correspondence map using linear least squares and our novel inverse spherical harmonic lighting model. The least squares residuals provide an unsupervised training signal that allows us to avoid artefacts common in the literature such as shrinking and conservative underfitting. Our approach uses a network that is 10ˆsmaller than parameter regression networks, significantly reduces sensitivity to image alignment and allows known camera calibration or multi-image constraints to be incorporated during inference. We achieve results competitive with state-of-the-art but without any auxiliary supervision used by previous methods.

show abstract

Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set

Cited by 643 publications

References 60 publications

Local Spherical Harmonics for Facial Shape and Albedo Estimation

Local Spherical Harmonics for Facial Shape and Albedo Estimation

Lightweight Photometric Stereo for Facial Details Recovery

“Look Ma, No Landmarks!” – Unsupervised, Model-Based Dense Face Alignment

Contact Info

Product

Resources

About