Differentiable Monte Carlo ray tracing through edge sampling

Li, Tzu‐Mao; Aittala, Miika; Durand, Frédo; Lehtinen, Jaakko

doi:10.1145/3272127.3275109

Cited by 354 publications

(325 citation statements)

References 49 publications

Supporting

Mentioning

320

Contrasting

Unclassified

Order By: Relevance

“…The differentiable projection function f (·) could be implemented via existing differentiable renderers [15], however we experimentally found them to be unstable when the parameters are far from their target, in addition to requiring significant computations for each scene. Instead, we implement f (·) by projecting each light source on the sphere onto a spherical gaussian using the mapping…”

Section: Training Step 1: Radius Color and Positionmentioning

confidence: 99%

Deep Parametric Indoor Lighting Estimation

Gardner

Hold-Geoffroy

Sunkavalli

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

126

156

View full text Add to dashboard Cite

We present a method to estimate lighting from a single image of an indoor scene. Previous work has used an environment map representation that does not account for the localized nature of indoor lighting. Instead, we represent lighting as a set of discrete 3D lights with geometric and photometric parameters. We train a deep neural network to regress these parameters from a single image, on a dataset of environment maps annotated with depth. We propose a differentiable layer to convert these parameters to an environment map to compute our loss; this bypasses the challenge of establishing correspondences between estimated and ground truth lights. We demonstrate, via quantitative and qualitative evaluations, that our representation and training scheme lead to more accurate results compared to previous work, while allowing for more realistic 3D object compositing with spatially-varying lighting.

show abstract

Section: Training Step 1: Radius Color and Positionmentioning

confidence: 99%

Deep Parametric Indoor Lighting Estimation

Gardner

Hold-Geoffroy

Sunkavalli

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

126

156

View full text Add to dashboard Cite

show abstract

“…'C*N(kS)' denotes convolution layers with N S × S filters with stride 2, followed by Batch Normalization and ReLU. BU (18,36) upsamples the response to produce 18 × 36 × 3 resolution environment map. Each 'ResBLK' contains Conv256(k3) -BN -ReLU -Conv256(k3) -BN, where 'ConvN(kS)' denotes convolution layers with N S×S filters of stride 1, 'BN' denoted Batch Normalization.…”

Section: Environment Map Estimatormentioning

confidence: 99%

“…': It first concatenates the responses of 'Enc', 'Normal ResBLKs' and 'Albedo ResBLKs' to produce a blob of spatial resolution 768 × 60 × 80. It is further processed by the following module: C256(k1) -C*256(k3) -C*128(k3) -C*3(k3) -BU (18,36) 'CN(kS)' denotes convolution layers with N S × S filters with stride 1, followed by Batch Normalization and ReLU. 'C*N(kS)' denotes convolution layers with N S × S filters with stride 2, followed by Batch Normalization and ReLU.…”

Section: Training On Synthetic Datamentioning

confidence: 99%

See 1 more Smart Citation

Neural Inverse Rendering of an Indoor Scene From a Single Image

Sengupta

Kim³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

136

View full text Add to dashboard Cite

Inverse rendering aims to estimate physical attributes of a scene, e.g., reflectance, geometry, and lighting, from image(s). Inverse rendering has been studied primarily for single objects or with methods that solve for only one of the scene attributes. We propose the first learning based approach that jointly estimates albedo, normals, and lighting of an indoor scene from a single image. Our key contribution is the Residual Appearance Renderer (RAR), which can be trained to synthesize complex appearance effects (e.g., inter-reflection, cast shadows, near-field illumination, and realistic shading), which would be neglected otherwise. This enables us to perform self-supervised learning on real data using a reconstruction loss, based on re-synthesizing the input image from the estimated components. We finetune with real data after pretraining with synthetic data. To this end we use physically-based rendering to create a largescale synthetic dataset, which is a significant improvement over prior datasets. Experimental results show that our approach outperforms state-of-the-art methods that estimate one or more scene attributes.

show abstract

“…HoloGAN first learns a 3D representation, which is then transformed to a target pose, projected to 2D features, and rendered to generate the final images ( Figure 2 right). Different from recent work that employs hand-crafted differentiable renderers [18,22,29,34,36,51,64], HoloGAN learns perspective projection and rendering of 3D features from scratch using a projection unit [40]. This novel architecture enables HoloGAN to learn 3D representations directly from natural images for which there are no good hand-crafted differentiable renderers.…”

Section: Introductionmentioning

confidence: 99%

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

Nguyen-Phuoc

Theis

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

338

397

View full text Add to dashboard Cite

Figure 1. HoloGAN learns to separate pose from identity (shape and appearance) only from unlabelled 2D images without sacrificing the visual fidelity of the generated images. All results shown here are sampled from HoloGAN for the same identities in each row but in different poses. AbstractWe propose a novel generative adversarial network (GAN) for the task of unsupervised learning of 3D representations from natural images. Most generative models rely on 2D kernels to generate images and make few assumptions about the 3D world. These models therefore tend to create blurry images or artefacts in tasks that require a strong 3D understanding, such as novel-view synthesis. HoloGAN instead learns a 3D representation of the world, and to render this representation in a realistic manner. Unlike other GANs, HoloGAN provides explicit control over the pose of generated objects through rigid-body transformations of the learnt 3D features. Our experiments show that using explicit 3D features enables HoloGAN to disentangle 3D pose and identity, which is further decomposed into shape and appearance, while still being able to generate images with similar or higher visual quality than other generative models. HoloGAN can be trained end-to-end from unlabelled 2D images only. In particular, we do not require pose labels, 3D shapes, or multiple views of the same objects. This shows that HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner.

show abstract

Differentiable Monte Carlo ray tracing through edge sampling

Abstract: times running from seconds to minutes depending on scene complexity and desired precision. We interface our differentiable ray tracer with the deep learning library PyTorch and show prototype applications in inverse rendering and the generation of adversarial examples for neural networks.

Cited by 354 publications

References 49 publications

Deep Parametric Indoor Lighting Estimation

Deep Parametric Indoor Lighting Estimation

Neural Inverse Rendering of an Indoor Scene From a Single Image

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

Contact Info

Product

Resources

About