Example-Based Modeling of Facial Texture from Deficient Data

Abstract

We present an approach to modeling ear-to-ear, highquality texture from one or more partial views of a face with possibly poor resolution and noise. Our approach is example-based in that we reconstruct texture with patches from a database composed of previously seen faces. A 3D morphable model is used to establish shape correspondence between the observed data across views and training faces. The database is built on the mesh surface by segmenting it into uniform overlapping patches. Texture patches are selected by belief propagation so as to be consistent with neighbors and with observations in an appropriate image formation model. We also develop a variant that is insensitive to light and camera parameters, and incorporate soft symmetry constraints. We obtain textures of higher quality for degraded views as small as 10 pixels wide, than a standard model fitted to non-degraded data. We further show applications to super-resolution where we substantially improve quality compared to a state-of-the-art algorithm, and to texture completion where we fill in missing regions and remove facial clutter in a photorealistic manner.

3D Morphable Face Models—Past, Present, and Future

“…In addition to global facial appearance models, there are also approaches that consider models of local skin variations. For example, Dessein et al [2015] use a texture model based on small overlapping patches that are extracted from a face database, and have presented a stochastic model that is able to synthesize freckles.…”

Appearance Controlled Face Texture Generation for Video Game Characters

“…Saito et al [2017] complete textures by analyzing the known parts of the texture and finding its relation to textures in a database to infer what the rest of the texture should be. A similar method is done by Dessein et al [2015] where they fill in occluded regions with similar known textures and then perform Poisson blending of the edges. Using image completion methods not specific to face textures, like the work by Iizuka et al [2017], has also been suggested.…”

Robust 3D Patch-Based Face Hallucination

“…Schumacher et al [36] added the blurring operator into the synthesis stage during 3D Morphable Model (3DMM) fitting [8] to restore degraded facial images. In the example-based approach of Dessein et al [14], the 3D mesh of the mean face was first segmented into uniformly overlapping patches. The LR pixels were then directly backprojected onto the corresponding LR vertices to apply belief propagation (BP) on the HR overlapping patches in a 3D MRF fashion [16].…”

“…Elevating the problem setup to the 3D level requires building a direct connection between the 3D shape s and the LR image z, of which the key challenge is to take into consideration the blurring kernel k. Dessein et al [14] just ignored it and back-projected the LR pixel values to the corresponding LR vertices. This oversimplified NN-like approach turns out to violate the image formation model [15] and struggles with real image data [34] in our evaluation.…”

“…Discussion It is worth noting that the newly published 3D MRF approach [14] used irregular 3D patches of fixed size segmented offline, which is cumbersome to further incorporate convolution with different blurring kernels.…”