Deep Parametric Indoor Lighting Estimation

Abstract

We present a method to estimate lighting from a single image of an indoor scene. Previous work has used an environment map representation that does not account for the localized nature of indoor lighting. Instead, we represent lighting as a set of discrete 3D lights with geometric and photometric parameters. We train a deep neural network to regress these parameters from a single image, on a dataset of environment maps annotated with depth. We propose a differentiable layer to convert these parameters to an environment map to compute our loss; this bypasses the challenge of establishing correspondences between estimated and ground truth lights. We demonstrate, via quantitative and qualitative evaluations, that our representation and training scheme lead to more accurate results compared to previous work, while allowing for more realistic 3D object compositing with spatially-varying lighting.

Image synthesis in visual machine learning

“…106 5.7 The proposed NeedleLight estimates a parametric lighting representation from a single scene image which is critical to many tasks such as virtual object insertion. Unlike previous methods that predict lighting in either frequency domain [2] (losing spatial localization) or spatial domain [3] (losing frequency information) only, we introduce a novel needlets basis which is capable of representing and estimating lighting accurately in both frequency and spatial domains. .…”

“…The recent learning-based works aim to estimate lighting from images by regressing representation parameters [3,59,60] or generating illumination maps [61,62].…”

“…Garon et al [2] estimate lighting by predicting spherical harmonic (SH) coefficients from a background image and local patch. Gardner et al [3] estimate the positions, intensities, and colours of light sources and reconstructs illumination maps with a spherical Gaussian function. On top of it, Li et al [63] represent illumination maps with multiple spherical Gaussian functions and regresses the corresponding Gaussian parameters for lighting estimation.…”

“…Following [3], we separate the light source by taking the 5% pixels that have the highest values within the HDR illumination map. The Light intensity I can then be determined by the summation of all the pixels within the light sources region, and the the ambient term A is further determined by the averaged pixel value within the remaining region (excluding light sources).…”

“…Lighting estimation has been tackled through direct generation of illumination maps [61,62,65] and regression of parameters of representative illumination functions such as spherical harmonics function [2,59] and spherical Gaussian function [3,60]. However, the functional representation methods struggle to regress accurate frequency information (especially high-frequency information) that often leads to inaccurate shading and shadow effects in relighting [2] or requires complex optimization steps [3]. Directly generating illumination maps can preserve some high-frequency information, but it can hardly recover other information of the light sources such as light directions and sizes [235].…”