Learning to reconstruct shape and spatially-varying reflectance from a single image

Abstract

Reconstructing shape and reflectance properties from images is a highly under-constrained problem, and has previously been addressed by using specialized hardware to capture calibrated data or by assuming known (or highly constrained) shape or reflectance. In contrast, we demonstrate that we can recover non-Lambertian, spatially-varying BRDFs and complex geometry belonging to any arbitrary shape class, from a single RGB image captured under a combination of unknown environment illumination and flash lighting. We achieve this by training a deep neural network to regress shape and reflectance from the image. Our network is able to address this problem because of three novel contributions: first, we build a large-scale dataset of procedurally generated shapes and real-world complex SVBRDFs that approximate real world appearance well. Second, single image inverse rendering requires reasoning at multiple scales, and we propose a cascade network structure that allows this in a tractable manner. Finally, we incorporate an in-network rendering layer that aids the reconstruction task by handling global illumination effects that are important for real-world scenes. Together, these contributions allow us to tackle the entire inverse rendering problem in a holistic manner and produce state-of-the-art results on both synthetic and real data.

Single image portrait relighting

“…There also exist neural networks [Calian et al 2018;Gardner et al 2017;Hold-Geoffroy et al 2017] that regress from an image (of a face, of an indoor environment, or of an outdoor environment respectively) to the environment that the image was illuminated by, but these approaches do not provide a solution for relighting. Li et al [2018] learns to regress from a single image of an object to a shape and spatially-varying BRDF that can then be used to relight that object, but requires a known and constrained illumination in the form of a front-facing flash, thereby preventing user manipulation of any existing natural illumination of the scene, and limiting the system's utility to situations in which the camera flash is the dominant light source and is not socially-disruptive. In this paper, we use a neural network to directly learn a function for relighting images in an end-to-end fashion, without explicitly modeling the geometry or the reflectance of the human face.…”

Deep Parametric Indoor Lighting Estimation

“…Barron and Malik [2] recover geometry, reflectance and illumination from a single image of an arbitrary object by enforcing hand-crafted priors on each component. Recently, deep learning-based methods have been proposed to recover illumination and material properties (along with, in some cases, geometry) from a single RGB image of an object [8,17,22,16]. These methods do not easily scale to large-scale indoor scenes where the illumination, geometry, and reflectance properties are significantly more complex.…”

Detecting Specular Reflections and Cast Shadows to Estimate Reflectance and Illumination of Dynamic Indoor Scenes

“…DATA-DRIVEN APPROACHES. Several end-to-end learning based approaches have been recently proposed to tackle the challenge of estimating illumination from HDR [18] or LDR [19] [20] [21] [22] images. In [18], they propose a method to infer high dynamic range illumination from a single panoramic photograph.…”

“…Nonetheless, reflectance estimation is not addressed. On the other hand, in [19], a CNN based framework is proposed to estimate shape and spatially varying reflectance, represented as diffuse albedo and specular roughness, from a single mobile phone photograph. In the considered context, scenes must be captured under controlled lighting conditions.…”