Path replay backpropagation

Abstract

Differentiable physically-based rendering has become an indispensable tool for solving inverse problems involving light. Most applications in this area jointly optimize a large set of scene parameters to minimize an objective function, in which case reverse-mode differentiation is the method of choice for obtaining parameter gradients. However, existing techniques that perform the necessary differentiation step suffer from either statistical bias or a prohibitive cost in terms of memory and computation time. For example, standard techniques for automatic differentiation based on program transformation or Wengert tapes lead to impracticably large memory usage when applied to physically-based rendering algorithms. A recently proposed adjoint method by Nimier-David et al. [2020] reduces this to a constant memory footprint, but the computation time for unbiased gradient estimates then becomes quadratic in the number of scattering events along a light path. This is problematic when the scene contains highly scattering materials like participating media. In this paper, we propose a new unbiased backpropagation algorithm for rendering that only requires constant memory, and whose computation time is linear in the number of scattering events (i.e., just like path tracing). Our approach builds on the invertibility of the local Jacobian at scattering interactions to recover the various quantities needed for reverse-mode differentiation. Our method also extends to specular materials such as smooth dielectrics and conductors that cannot be handled by prior work.

Curriculum Learning for ab initio Deep Learned Refractive Optics

“…Neither of these approaches provides a satisfactory solution to large-scale, high resolution deep lens design of complex optical systems. Instead, we recalculate the ray-tracing simulation during backpropagation, as in [30]. First, we perform ray-tracing to simulate the sensor image without tracking gradient information.…”

Dressi: A Hardware‐Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization

“…Path‐space DR [ZMY ∗ 20] shows better efficiency using the new Monte Carlo estimators. Path replay backpropagation [VSJ21] proposes a ray tracing‐based mega‐kernel generation that maintains linear time complexity with constant memory usage. Ray tracing‐based methods focus on photorealistic results and accurate backpropagation.…”

“…Although they have shown remarkable speedups recently, they still require exceedingly high computational costs for low‐end devices lacking hardware ray tracers. Some code optimization techniques that reduce heavy computations on high‐end hardware [NVZJ19; VSJ21] are related to our stage packing process. However, ours differs from existing techniques because it is designed to adapt to varying graphics hardware constraints.…”

Path-space differentiable rendering

“…Our technique presented above is closely related to some recent works [61,84], which formulate image-loss gradients as solutions of an adjoint transport problem. We will show that some key results in these work can be considered a specific realization of Eq.…”