Sparse Photometric 3D Face Reconstruction Guided by Morphable Models

Abstract

Figure 1: Sample results using our sparse PS reconstruction. By using just 5 input images (left), our method can recover very high quality 3D face geometry with fine geometric details. AbstractWe present a novel 3D face reconstruction technique that leverages sparse photometric stereo (PS) and latest advances on face registration/modeling from a single image. We observe that 3D morphable faces approach[15] provides a reasonable geometry proxy for light position calibration. Specifically, we develop a robust optimization technique that can calibrate per-pixel lighting direction and illumination at a very high precision without assuming uniform surface albedos. Next, we apply semantic segmentation on input images and the geometry proxy to refine hairy vs. bare skin regions using tailored filters. Experiments on synthetic and real data show that by using a very small set of images, our technique is able to reconstruct fine geometric details such as wrinkles, eyebrows, whelks, pores, etc, comparable to and sometimes surpassing movie quality productions.

Photo-Realistic Facial Details Synthesis From Single Image

“…Face Scan Capture. To acquire training datasets, we implement a small-scale facial capture system similar to [13] and further enhance photometric stereo with multiview stereo: the former can produce high quality local details but is subject to global deformation whereas the latter shows good performance on low frequency geometry and can effectively correct deformation.…”

“…Instead, we observe that despite large scale variations on different faces, local texture details present strong similarities even if the faces appear vastly different. Hence we adopt the idea from [9,13] to enhance our network generalization by training the network with texture/displacement patches of 256 × 256 resolution. We model the displacement using PCA, where each patch is a linear combination of 64 basis patches.…”

Local Spherical Harmonics for Facial Shape and Albedo Estimation

“…SfS, which is a technique for inferring the normal map of a given image, has been widely adopted for enhancing high-frequency details with an initial estimate of a 3D model. After using various methods to compute an initial model, such as multi-view stereo [15], depth [16], or template-based methods [13], the surface normal, illumination, and albedo are inferred to reconstruct a point cloud and corresponding mesh. For human faces, SfS-based methods can successfully recover 3D geometry from a set of unconstrained [14] or studio level [13], [28] images.…”

“…After using various methods to compute an initial model, such as multi-view stereo [15], depth [16], or template-based methods [13], the surface normal, illumination, and albedo are inferred to reconstruct a point cloud and corresponding mesh. For human faces, SfS-based methods can successfully recover 3D geometry from a set of unconstrained [14] or studio level [13], [28] images. In [14], a 3D surface corresponding to computed point normals was generated by deforming a template shape using the Laplacian surface deformation technique [29].…”

“…However, such methods can only reconstruct the smooth parts of facial geometry and fail to reconstruct the high-frequency details such as beard, pore, and wrinkles. In recent years, several methods have been proposed to recover high-frequency details from the smooth geometry by using the shape from shading (SfS) technique [13], [14]. This technique is widely adopted for computing the 3D geometry of highly curved regions based on the illumination and reflectance model for shading variation [15], [16].…”