Realistic inverse lighting from a single 2D image of a face, taken under unknown and complex lighting

Abstract

In this paper, we address a difficult inverse rendering problem with many unknowns: a single 2D input image of an unknown face in an unknown environment, taken under unknown conditions. First, the geometry and texture of the face are estimated from the input image, using a 3D Morphable Model. In a second step, considering the superposition principle for light, we estimate the light source intensities as optimized non-negative weights for a linear combination of a synthetic illumination cone for that face. Each image of the illumination cone is lighted by one directional light, considering non-lambertian reflectance and non-convex geometry. Modeling the lighting separately from the face model enhances the face modeling and analysis, provides information about the environment of the face, and facilitates realistic rendering of the face in novel pose and lighting.

Learning Illumination from Diverse Portraits

“…With the appropriate lighting basis and re ectance assumption, the problem was reduced to inverting a linear system of equations. e linearity of light transport was similarly leveraged in follow-up work to estimate lighting from faces (Shahlaei and Blanz 2015;Shim 2012), including for real-time AR (Knorr and Kurz 2014), but these approaches estimated either a small number of point light sources or again used a low frequency 2 nd order SH lighting basis. Specular re ections from the eyes of portrait subjects have been leveraged to estimate higher frequency illumination, but as the re ections of bright light sources are likely to be clipped, the recovery of the full dynamic range of natural illumination is challenging to recover from a single exposure image (Nishino and Nayar 2004).…”

From Faces to Outdoor Light Probes

“…In graphics, similar ideas have been used in face swapping [BKD * 08, DSJ * 11]. While accurate lighting estimation from a face image has already been explored [Shi12, KK14,SB15], previous work has focused solely on recovering low-frequency, low dynamic range indoor lighting; performance has been evaluated only qualitatively by relighting a sphere. While [NN04] estimate illumination conditions from a single image of an eye, reduced pixel resolution is an important limitation in this case.…”

Label Denoising Adversarial Network (LDAN) for Inverse Lighting of Faces

“…Blanz and Vetter [9] proposed to estimate the ambient and directional light as a byproduct of fitting 3D Morphable Models (3DMM) to a single face image. Since then, several 3DMM based methods were proposed [2,25,17,30,33]. The performance of these methods rely on a good 3DMM of faces.…”