Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning

Abstract

Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers [29,19], which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images.The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-thearts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised singleview reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers. Code is available at https://github.com/ ShichenLiu/SoftRas.

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

“…A module of boundary refinement is then adopted to refine the boundary conditions. renderer [19,16] have proposed to train a mesh generator based on rendering loss, eliminating the need of 3D supervision. However, no prior approaches can dynamically modify the topology of the template mesh, while we propose the first topology modification network that is able to generate meshes with complex topologies from a genus-0 3D model.…”

Differentiable surface splatting for point-based geometry processing

“…Liu et al [2018] propose Paparazzi, an analytic DR for mesh geometry processing using image filters. In concurrent work, Petersen et al [2019] present Pix2Vex, a C ∞ differentiable renderer via soft blending schemes of nearby triangles, and Liu et al [2019] introduce Soft Rasterizer, which renders and aggregates the probabilistic maps of mesh triangles, allowing flowing gradients from the rendered pixels to the occluded and far-range vertices. All these generic DR frameworks rely on mesh representation of the scene geometry.…”

“…Advanced learning-based methods for point set processing are currently emerging, encouraged by the success of deep learning. Based on PointNet [Qi et al 2017], PCPNET [Guerrero et al 2018] and PointCleanNet [Rakotosaona et al 2019] [Liu et al 2018] has limitation in updating the vertex positions in directions orthogonal their face normals, thus can not alter the silhouette of shapes; Soft Rasterizer [Liu et al 2019] and Pix2Vex [Petersen et al 2019] can pass gradient to occluded vertices, through blurred edges and transparent faces. All mesh renderers do not consider the normal field directly and cannot modify mesh topology.…”

“…OpenDR [Loper and Black 2014] computes gradients only in screen space from a small set of pixels near the boundary, which is conceptually less accurate than our definition. SoftRasterizer [Liu et al 2019] alters the forward rendering to make the rasterization step inherently differentiable; this leads to impeded rendering quality and relies on hyper-parameters to control the differentiability (i.e., support of non-zero gradient). The work related most closely to our approach in terms of gradient definition is the neural mesh renderer (NMR) [Kato et al 2018].…”

“…Comparison to filter-based gradient approximation. Alternatively, related to SoftRasterizer [Liu et al 2019] and Pix2Vex [Petersen et al 2019], one can define the gradient of the discontinuous function Φ x,k by replacing it with a C ∞ function, e.g., a radial basis function (RBF). This is a seemingly natural choice for EWA-based point rendering, since each point is represented as a RBF in the forward pass.…”