Fast Spatially-Varying Indoor Lighting Estimation

Abstract

We propose a real-time method to estimate spatiallyvarying indoor lighting from a single RGB image. Given an image and a 2D location in that image, our CNN estimates a 5th order spherical harmonic representation of the lighting at the given location in less than 20ms on a laptop mobile graphics card. While existing approaches estimate a single, global lighting representation or require depth as input, our method reasons about local lighting without requiring any geometry information. We demonstrate, through quantitative experiments including a user study, that our results achieve lower lighting estimation errors and are preferred by users over the state-of-the-art. Our approach can be used directly for augmented reality applications, where a virtual object is relit realistically at any position in the scene in real-time.

Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

“…The results are illustrated in Table 3. The results show that our scores are approximate 0.2 higher than Garon et al [20] and are much higher than Tsai et al [9] at both criteria. It indicates that with reflections and improved shadows, our method can bring more reality to the compositing results.…”

“…Figures 11 and 12 compare our method with pure lighting reconstruction systems like DeepLight [18] and the work of Garon et al [20] as well as harmonization method like Tsai et al [9]. To make Tsai et al [9] comparable with ours, we render the foreground objects with a constant uniform lighting and run their network to get the results.…”

“…The age range is 15-53 (average: 30.7, median: 26.5) in the valid questionaires. We create a test set of 15 real photographs (11 indoor scenes and 4 outdoor scenes) and show the subjects compositing results (see Figure S3 in Appendix A) of our method, Tsai et al [9] and Garon et al [20]. The compositing images of each scene were randomly shuffled when presented to subjects.…”

“…The reconstruction can be performed by photographing light probes [11,12], labeling lights interactively [11], or be automated with an optimization process [13][14][15]. Deep neural networks can also learn relevant information from photographs, including lighting [16][17][18][19][20], geometry and albedo [21], or even SVBRDF [22][23][24][25]. Regardless of how the real scene is reconstructed, though, final compositing still comes to differential rendering [12,13,18,[26][27][28].…”

Detecting Specular Reflections and Cast Shadows to Estimate Reflectance and Illumination of Dynamic Indoor Scenes

“…In fact, we refine k s in order to take account for the approximation introduced in equation (19) as the reflection and view vectors may not be at optimal alignment within mask H. Also, since our light sources are recovered from a discrete set of hypothetical point lights (section 3.3), a trade-off between fine sampling and real-time constraints must be considered. Hence, we initially define a coarse sampling (1176 point lights with a sampling step of 20cm) and refine the positions using equation (20).…”