Neural Illumination: Lighting Prediction for Indoor Environments

Song, Shuran; Funkhouser, Thomas

doi:10.1109/cvpr.2019.00708

Cited by 115 publications

(149 citation statements)

References 28 publications

Supporting

Mentioning

149

Contrasting

Order By: Relevance

“…Legendre et al extend this work to mobile applications and obtain better results by using a collection of videos as training data [14]. Song et al [23] use Matter-port3D [5] dataset and a novel warping procedure in order to support multiple insertion points. We improve on their work by training an end-to-end neural network to predict discrete parametric 3D lights with 3D position, area, color and intensity.…”

Section: Related Workmentioning

confidence: 99%

Deep Parametric Indoor Lighting Estimation

Gardner

Hold-Geoffroy

Sunkavalli

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

126

156

View full text Add to dashboard Cite

We present a method to estimate lighting from a single image of an indoor scene. Previous work has used an environment map representation that does not account for the localized nature of indoor lighting. Instead, we represent lighting as a set of discrete 3D lights with geometric and photometric parameters. We train a deep neural network to regress these parameters from a single image, on a dataset of environment maps annotated with depth. We propose a differentiable layer to convert these parameters to an environment map to compute our loss; this bypasses the challenge of establishing correspondences between estimated and ground truth lights. We demonstrate, via quantitative and qualitative evaluations, that our representation and training scheme lead to more accurate results compared to previous work, while allowing for more realistic 3D object compositing with spatially-varying lighting.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Parametric Indoor Lighting Estimation

Gardner

Hold-Geoffroy

Sunkavalli

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

126

156

View full text Add to dashboard Cite

show abstract

“…The reconstruction can be performed by photographing light probes [11,12], labeling lights interactively [11], or be automated with an optimization process [13][14][15]. Deep neural networks can also learn relevant information from photographs, including lighting [16][17][18][19][20], geometry and albedo [21], or even SVBRDF [22][23][24][25]. Regardless of how the real scene is reconstructed, though, final compositing still comes to differential rendering [12,13,18,[26][27][28].…”

Section: Related Workmentioning

confidence: 99%

Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

Shen

Hou

et al. 2021

Sci. China Inf. Sci.

View full text Add to dashboard Cite

We present neural compositing, a deep-learning based method for augmented reality rendering, which uses convolutional neural networks to composite rendered layers of a virtual object with a real photograph to emulate shadow and reflection effects. The method starts from estimating the lighting and roughness information from the photograph using neural networks, renders the virtual object with a virtual floor into color, shadow and reflection layers by applying the estimated lighting, and finally refines the reflection and shadow layers using neural networks and blends them with the color layer and input image to yield the output image. We assume low-frequency lighting environments and adopt PRT (precomputed radiance transfer) for layer rendering, which makes the whole pipeline differentiable and enables fast end-to-end network training with synthetic scenes. Working on a single photograph, our method can produce realistic reflections in a real scene with spatially-varying material and cast shadows on background objects with unknown geometry and material at real-time frame rates.

show abstract

“…Spatially-variant lighting information can be traditionally extracted using physical probes [6,21], and more recently estimated with deep neural networks [7,8,28,33]. For example, Debevec et al demonstrated that spatially-variant lighting can be effectively estimated by using reflective sphere light probes to extrapolate camera views.…”

Section: Introductionmentioning

confidence: 99%

“…As we later observe that XiheNet models trained with LDR-based datasets lead to better visual effects than that of HDR-based datasets; for the remainder of the paper, we will report results using XiheNet trained on LDR-based datasets.Xihe outputs SH coefficients as an omnidirectional representation of environment lighting at a single world position for rendering. If directly using image-based lighting estimation models[8,28], one needs to post-process to correctly orient estimated SH coefficients since the 3D world orientation cannot be represented on the image input. Our XiheNet guarantees the orientation constant[33] by explicitly considers the world space point cloud and estimates SH coefficients at the same orientation.…”

mentioning

confidence: 99%

Xihe

Zhao

Guo

2021

Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services

View full text Add to dashboard Cite

Omnidirectional lighting provides the foundation for achieving spatially-variant photorealistic 3D rendering, a desirable property for mobile augmented reality applications. However, in practice, estimating omnidirectional lighting can be challenging due to limitations such as partial panoramas of the rendering positions, and the inherent environment lighting and mobile user dynamics. A new opportunity arises recently with the advancements in mobile 3D vision, including built-in high-accuracy depth sensors and deep learning-powered algorithms, which provide the means to better sense and understand the physical surroundings. Centering the key idea of 3D vision, in this work, we design an edge-assisted framework called Xihe to provide mobile AR applications the ability to obtain accurate omnidirectional lighting estimation in real time.Specifically, we develop a novel sampling technique that efficiently compresses the raw point cloud input generated at the mobile device. This technique is derived based on our empirical analysis of a recent 3D indoor dataset and plays a key role in our 3D vision-based lighting estimator pipeline design. To achieve the realtime goal, we develop a tailored GPU pipeline for on-device point cloud processing and use an encoding technique that reduces network transmitted bytes. Finally, we present an adaptive triggering strategy that allows Xihe to skip unnecessary lighting estimations and a practical way to provide temporal coherent rendering integration with the mobile AR ecosystem. We evaluate both the lighting estimation accuracy and time of Xihe using a reference mobile application developed with Xihe's APIs. Our results show that Xihe takes as fast as 20.67ms per lighting estimation and achieves 9.4% better estimation accuracy than a state-of-the-art neural network. CCS CONCEPTS• Computing methodologies → Mixed / augmented reality; • Human-centered computing → Ubiquitous and mobile computing systems and tools; • Computer systems organization → Distributed architectures.

show abstract

Neural Illumination: Lighting Prediction for Indoor Environments

Cited by 115 publications

References 28 publications

Deep Parametric Indoor Lighting Estimation

Deep Parametric Indoor Lighting Estimation

Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

Xihe

Contact Info

Product

Resources

About