Learning Lightprobes for Mixed Reality Illumination

Abstract

Figure 1: Coherent Illumination. The real scene consists of an action figure of The Hulk and a toy car. To demonstrate the result of our system, we place a 3D scan next to the real action figure and display it using Mixed Reality: (left) The 3D reconstruction is rendered without real world light estimation. (right) Our system estimates the current lighting from a single input image. The estimated lighting is used to illuminate the 3D reconstruction. Note that we only register the real world lighting and do not consider any camera effects such as exposure or blur. ABSTRACTThis paper presents the first photometric registration pipeline for Mixed Reality based on high quality illumination estimation by convolutional neural network (CNN) methods. For easy adaptation and deployment of the system, we train the CNN using purely synthetic images and apply them to real image data. To keep the pipeline accurate and efficient, we propose to fuse the light estimation results from multiple CNN instances, and we show an approach for caching estimates over time. For optimal performance, we furthermore explore multiple strategies for the CNN training. Experimental results show that the proposed method yields highly accurate estimates for photo-realistic augmentations.

DeepLight: light source estimation for augmented reality using deep learning

“…We can also use a human face as a light probe to capture illumination from the front-facing camera of a mobile phone [20]. Recently, Mandl et al showed that it is possible to utilize an arbitrary object as a light probe [22]. In their method, a series of neural networks are trained for a given light probe object.…”

“…Typically, real-world illumination can be estimated using a passive or active light probe positioned in a scene [19,22]. Ideally, light sources would be estimated without the light probe to avoid the necessity of undesirable objects in the scene [9,11].…”

“…In order to maintain the convergence of training with increasing network depth and to avoid a vanishing/exploding gradients problem, we design our network using residual blocks of convolutional layers [12]. Previous research showed that it is possible to calculate diffuse lighting in form of an omnidirectional image by a neural network [9,22]. In the previous work, a neural network was used to calculate the image-to-image relationship between an input image and the estimated illumination.…”

Learning to Estimate Indoor Lighting from 3D Objects

“…Gruber et al [17] model the Radiance Transfer Function of an entire scene from its captured geometry, but it requires from 50 to 200 ms per frame. To reduce the time complexity, and bearing similarities to our approach, Mandl et al [31] uses deep learning to estimate the SH coefficients from an object. In our case, we do not learn SH coefficients but instead learn to map an image to the latent space of indoor lighting.…”

“…Estimating lighting from images enables a wide range of possible applications, ranging from the realistic insertion of virtual content in augmented reality [11,31,23], shadow or highlight removal [22], image matching [38], appearance transfer [28] or reflectance and/or geometry estimation [29] to name just a few. However, estimating light from an image is a challenging problem.…”