A Morphable Face Albedo Model

Smith, William A. P.; Seck, Alassane; Dee, Hannah; Tiddeman, Bernard; Tenenbaum, Joshua B.; Egger, Bernhard

doi:10.1109/cvpr42600.2020.00506

Cited by 47 publications

(19 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If the vertex is not visible in one view, i.e., the corresponding visibility score is smaller than 0, the score is set to zero. We then set a threshold distance of the occlusion boundary following AlbedoMM (Smith et al 2020). If the vertex is projected within the occlusion boundary's threshold distance, we set its visibility score to zero to avoid sampling background onto the mesh.…”

Section: Masking and Stitchingmentioning

confidence: 99%

Towards High-Fidelity Face Self-Occlusion Recovery via Multi-View Residual-Based GAN Inversion

Chen

Han

Shan

2022

AAAI

View full text Add to dashboard Cite

Face self-occlusions are inevitable due to the 3D nature of the human face and the loss of information in the projection process from 3D to 2D images. While recovering face self-occlusions based on 3D face reconstruction, e.g., 3D Morphable Model (3DMM) and its variants provides an effective solution, most of the existing methods show apparent limitations in expressing high-fidelity, natural, and diverse facial details. To overcome these limitations, we propose in this paper a new generative adversarial network (MvInvert) for natural face self-occlusion recovery without using paired image-texture data. We design a coarse-to-fine generator for photorealistic texture generation. A coarse texture is computed by inpainting the invisible areas in the photorealistic but incomplete texture sampled directly from the 2D image using the unrealistic but complete statistical texture from 3DMM. Then, we design a multi-view Residual-based GAN Inversion, which re-renders and refines multi-view 2D images, which are used for extracting multiple high-fidelity textures. Finally, these high-fidelity textures are fused based on their visibility maps via Poisson blending. To perform adversarial learning to assure the quality of the recovered texture, we design a discriminator consisting of two heads, i.e., one for global and local discrimination between the recovered texture and a small set of real textures in UV space, and the other for discrimination between the input image and the re-rendered 2D face images via pixel-wise, identity, and adversarial losses. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods in face self-occlusion recovery under unconstrained scenarios.

show abstract

Section: Masking and Stitchingmentioning

confidence: 99%

Towards High-Fidelity Face Self-Occlusion Recovery via Multi-View Residual-Based GAN Inversion

Chen

Han

Shan

2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Similar works have also modeled complete head topologies [39], [40], [41], [42]. Finally, the recent Morphable Face Albedo Model [10] separately models diffuse and specular albedo with a PCA model.…”

Section: Facial Geometry and Texture Estimationmentioning

confidence: 99%

“…In the years that followed, 3DMM fitting was extended so that it could use a PCA model on robust features, i.e., Histogram of Oriented Gradients (HOGs) [8], for representing facial texture [9], with improved results in "in-the-wild" images. The recently proposed, Morphable Face Albedo model [10] additionally reconstructs diffuse and specular albedo with PCA. Nevertheless, these methods cannot reconstruct highresolution facial textures.…”

Section: Introductionmentioning

confidence: 99%

AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs

Lattas,

Moschoglou,

Ploumpis

et al. 2021

Preprint

View full text Add to dashboard Cite

Over the last years, with the advent of Generative Adversarial Networks (GANs), many face analysis tasks have accomplished astounding performance, with applications including, but not limited to, face generation and 3D face reconstruction from a single "in-the-wild" image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from "in-the-wild" images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this paper, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single "in-the-wild" image. To achieve this, we capture a large dataset of facial shape and reflectance, which we have made public. Moreover, we define a fast and photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular reflectance components from a mesh and texture with baked illumination, scanned or reconstructed with a 3DMM fitting method. As we demonstrate in a series of qualitative and quantitative experiments, our method outperforms the existing arts by a significant margin and reconstructs authentic, 4K by 6K-resolution 3D faces from a single low-resolution image, that are ready to be rendered in various applications and bridge the uncanny valley.

show abstract

“…More specifically, the encoder regresses the scale parameters, shape parameters, expression parameters, and other parameters for rendering, such as albedo parameters, illumination parameters, pose parameters, and camera parameters. In the decoder part, we have four components, each of which is to be trained in this stage: (1) The trained shape basis of SFM, (2) The expression basis D exp from bfm2017 [16], (3)the albedo basis D albedo from [45], (4) the rendering layer takes the geometric, albedo, illumination, pose parameter, and camera parameter and renders 224×224 RGB images, which is based on Pytorch3d [36]. The illumination model is a spherical harmonic illumination model.…”

Section: Learning Frameworkmentioning

confidence: 99%

Sphere Face Model:A 3D Morphable Model with Hypersphere Manifold Latent Space

Jiang¹,

Jin²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

3D Morphable Models (3DMMs) are generative models for face shape and appearance. However, the shape parameters of traditional 3DMMs satisfy the multivariate Gaussian distribution while the identity embeddings satisfy the hypersphere distribution, and this conflict makes it challenging for face reconstruction models to preserve the faithfulness and the shape consistency simultaneously. To address this issue, we propose the Sphere Face Model(SFM), a novel 3DMM for monocular face reconstruction, which can preserve both shape fidelity and identity consistency. The core of our SFM is the basis matrix which can be used to reconstruct 3D face shapes, and the basic matrix is learned by adopting a two-stage training approach where 3D and 2D training data are used in the first and second stages, respectively. To resolve the distribution mismatch, we design a novel loss to make the shape parameters have a hyperspherical latent space. Extensive experiments show that SFM has high representation ability and shape parameter space's clustering performance. Moreover, it produces fidelity face shapes, and the shapes are consistent in challenging conditions in monocular face reconstruction.

show abstract

A Morphable Face Albedo Model

Abstract: Figure 1: First 3 principal components of our statistical diffuse (left) and specular (middle) albedo models. Both are visualised in linear sRGB space. Right: rendering of the combined model under frontal illumination in nonlinear sRGB space.

Cited by 47 publications

References 33 publications

Towards High-Fidelity Face Self-Occlusion Recovery via Multi-View Residual-Based GAN Inversion

Towards High-Fidelity Face Self-Occlusion Recovery via Multi-View Residual-Based GAN Inversion

AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs

Sphere Face Model:A 3D Morphable Model with Hypersphere Manifold Latent Space

Contact Info

Product

Resources

About