Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction

Abstract

Fig. 1. Our method takes a single image as the input and predicts the full and spatially-varying indoor illumination that can be used to generate consistent shading and realistic shadows after inserting virtual objects into the image.Lighting prediction from a single image is becoming increasingly important in many vision and augmented reality (AR) applications in which shading and shadow consistency between virtual and real objects should be guaranteed. However, this is a notoriously ill-posed problem, especially for indoor scenarios, because of the complexity of indoor luminaires and the limited information involved in 2D images. In this paper, we propose a graph learning-based framework for indoor lighting estimation. At its core is a new lighting model (dubbed DSGLight) based on depth-augmented Spherical Gaussians (SG) and a Graph Convolutional Network (GCN) that infers the new lighting representation from a single LDR image of limited field-ofview. Our lighting model builds 128 evenly distributed SGs over the indoor panorama, where each SG encoding the lighting and the depth around that node. The proposed GCN then learns the mapping from the input image to DSGLight. Compared with existing lighting models, our DSGLight encodes both direct lighting and indirect environmental lighting more faithfully and compactly. It also makes network training and inference more stable. The estimated depth distribution enables temporally stable shading and shadows under spatially-varying lighting. Through thorough experiments, we show that our method obviously outperforms existing methods both qualitatively and quantitatively.CCS Concepts: • Computing methodologies → Mixed / augmented reality; Scene understanding.

Editable Indoor Lighting Estimation

“…Similar to us, GMlight [31] backprojects the spherical gaussians to an estimated 3D model of the scene. This is further extended in [1] by the use of graph neural networks, and in [32] through the use of spherical wavelets dubbed "needlets".…”