Light stage super-resolution

Sun, Tongqing; Xu, Zexiang; Zhang, Xiuming; Fanello, Sean; Rhemann, Christoph; Debevec, Paul; Tsai, Yun-Ta; Barron, Jonathan T.; Ramamoorthi, Ravi

doi:10.1145/3414685.3417821

Cited by 42 publications

(9 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The staggering advances in machine learning in the last decade have also had a profound effect on imagebased relighting [Debevec et al 2000], enabling new capabilities and improving quality [Bemana et al 2020;Ren et al 2015;Xu et al 2018]. Deep learning has subsequently been applied to more specialized relighting tasks for portraits Meka et al 2019;Sun et al 2019Sun et al , 2020, full bodies [Guo et al 2019;Kanamori and Endo 2018;Meka et al 2020;Yeh et al 2022;Zhang et al 2021a], and outdoor scenes [Griffiths et al 2022;Meshry et al 2019;. It is unclear how to extend these methods to handle scenes that contain objects with ill-defined shapes (e.g., fur) and translucent and specular materials.…”

Section: Related Workmentioning

confidence: 99%

Relighting Neural Radiance Fields with Shadow and Highlight Hints

Zeng

Chen

Dong

et al. 2023

Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings

View full text Add to dashboard Cite

This paper presents a novel neural implicit radiance representation for free viewpoint relighting from a small set of unstructured photographs of an object lit by a moving point light source different from the view position. We express the shape as a signed distance function modeled by a multi layer perceptron. In contrast to prior relightable implicit neural representations, we do not disentangle the different light transport components, but model both the local and global light transport at each point by a second multi layer perceptron that, in addition, to density features, the current position, the normal (from the signed distance function), view direction, and light position, also takes shadow and highlight hints to aid the network in modeling the corresponding high frequency light transport effects. These hints are provided as a suggestion, and we leave it up to the network to decide how to incorporate these in the final relit result. We demonstrate and validate our neural implicit representation on synthetic and real scenes exhibiting a wide variety of shapes, material properties, and global illumination light transport. CCS CONCEPTS• Computing methodologies → Image-based rendering; Reflectance modeling.

show abstract

Section: Related Workmentioning

confidence: 99%

Relighting Neural Radiance Fields with Shadow and Highlight Hints

Zeng

Chen

Dong

et al. 2023

Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings

View full text Add to dashboard Cite

show abstract

“…Furthermore, lighting and material information requirements can be relaxed using Inverse Path Tracing [ALKN19], however, a full and accurate scene geometry is still essential for high quality rendering. If very high quality results are required (e. g., for film production), sophisticated capture sets ups such as the Light Stage [DHT∗00, WGT∗05, MHP∗19, GLD∗19, SXZ∗20] can be employed. Whilst such methods give impressive results, they require expensive rigs and trained professionals to operate them.…”

Section: Previous Workmentioning

confidence: 99%

OutCast: Outdoor Single‐image Relighting with Cast Shadows

Griffiths

Ritschel

Philip

2022

Computer Graphics Forum

View full text Add to dashboard Cite

We propose a relighting method for outdoor images. Our method mainly focuses on predicting cast shadows in arbitrary novel lighting directions from a single image while also accounting for shading and global effects such the sun light color and clouds. Previous solutions for this problem rely on reconstructing occluder geometry, e. g., using multi‐view stereo, which requires many images of the scene. Instead, in this work we make use of a noisy off‐the‐shelf single‐image depth map estimation as a source of geometry. Whilst this can be a good guide for some lighting effects, the resulting depth map quality is insufficient for directly ray‐tracing the shadows. Addressing this, we propose a learned image space ray‐marching layer that converts the approximate depth map into a deep 3D representation that is fused into occlusion queries using a learned traversal. Our proposed method achieves, for the first time, state‐of‐the‐art relighting results, with only a single image as input. For supplementary material visit our project page at: http://dgriffiths.uk/outcast.

show abstract

“…To alleviate this issue, we adopt the pooling technique introduced by Sun et al [55]. We replace the mean µ and variance v of {f i } N i=1 with a weighted mean µ w and variance v w , weighted so as to reduce the influence of the furthest images in the working set.…”

Section: Volume Density Predictionmentioning

confidence: 99%

IBRNet: Learning Multi-View Image-Based Rendering

Wang¹,

Wang²,

Genova³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations (3D spatial locations and 2D viewing directions), drawing appearance information on the fly from multiple source views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering (IBR), and allows us to render high-resolution imagery. Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multiview posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods. 1

show abstract

Light stage super-resolution

Abstract: I 1 and I 2 , captured light stage images for adjacent lights ℓ 1 and ℓ 2 (b) Blended images (I 1 + I 2)/2 (c) Our rendering I((ℓ 1 + ℓ 2)/2)

Cited by 42 publications

References 32 publications

Relighting Neural Radiance Fields with Shadow and Highlight Hints

Relighting Neural Radiance Fields with Shadow and Highlight Hints

OutCast: Outdoor Single‐image Relighting with Cast Shadows

IBRNet: Learning Multi-View Image-Based Rendering

Contact Info

Product

Resources

About