RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Abstract

Traditional computer graphics rendering pipelines are designed for procedurally generating 2D images from 3D shapes with high performance. The nondifferentiability due to discrete operations (such as visibility computation) makes it hard to explicitly correlate rendering parameters and the resulting image, posing a significant challenge for inverse rendering tasks. Recent work on differentiable rendering achieves differentiability either by designing surrogate gradients for non-differentiable operations or via an approximate but differentiable renderer. These methods, however, are still limited when it comes to handling occlusion, and restricted to particular rendering effects. We present RenderNet, a differentiable rendering convolutional network with a novel projection unit that can render 2D images from 3D shapes. Spatial occlusion and shading calculation are automatically encoded in the network. Our experiments show that RenderNet can successfully learn to implement different shaders, and can be used in inverse rendering tasks to estimate shape, pose, lighting and texture from a single image.Recent work in differentiable rendering achieves differentiability in various ways. Loper and Black [2] propose an approximate renderer which is differentiable. Kato et al. [3] achieve differentiability by proposing an approximate gradient for the rasterization operation. Recent work on image-based reconstruction uses differentiable projections of 3D objects onto silhouette masks as a surrogate for a rendered image of the objects [4, 5]. Wu et al. [6] and Tulsiani et al. [7] derive differentiable projective 32nd Conference on Neural Information Processing Systems (NeurIPS 2018),

Deep Direct Volume Rendering: Learning Visual Feature Mappings From Exemplary Images

“…The specific combination of volume rendering with machine learning has been addressed in recent literature: [32] introduced RenderNet, a deep convnet that performs differentiable rendering of voxelized 3D shapes. Their proposed network consists of a 3D and a 2D convolutional part connected by a novel projection unit which combines the features along the viewing ray with an MLP.…”

“…6, are progressing in complexity by replacing more of the functions with deep neural networks. For comparison, we also include Render-Net [32] as a baseline in our experiments as, among previously published works, this architecture is most closely related to our concepts. RenderNet resamples the input volume to view space using the perspective camera transform, such that the X-axis of the resampled volume corresponds to the ray directions and the YZ-axes correspond to the image coordinates.…”

GNeRF: GAN-based Neural Radiance Field without Posed Camera

“…But these methods operate on 2D space of pixels, ignoring the 3d structure of our natural scene. 3Daware image synthesis correlates 3D model with 2D images, enabling explicit modification of 3D model [33,13,13,34,39,35,5,4]. Earlier 3D-aware image synthesis methods like RenderNet [33] introduces rendering convolutional networks with a projection unit that can render 2D images from 3D shapes.…”

Shadow Art Revisited: A Differentiable Rendering Based Approach

“…Several methods operate on voxel grids [7,12,15,20]. Paschalidou et al [15] and Tulsiani et al [20] propose a probabilistic ray potential formulation.…”