Systematically differentiating parametric discontinuities

Abstract

Emerging research in computer graphics, inverse problems, and machine learning requires us to differentiate and optimize parametric discontinuities. These discontinuities appear in object boundaries, occlusion, contact, and sudden change over time. In many domains, such as rendering and physics simulation, we differentiate the parameters of models that are expressed as integrals over discontinuous functions. Ignoring the discontinuities during differentiation often has a significant impact on the optimization process. Previous approaches either apply specialized hand-derived solutions, smooth out the discontinuities, or rely on incorrect automatic differentiation. We propose a systematic approach to differentiating integrals with discontinuous integrands, by developing a new differentiable programming language. We introduce integration as a language primitive and account for the Dirac delta contribution from differentiating parametric discontinuities in the integrand. We formally define the language semantics and prove the correctness and closure under the differentiation, allowing the generation of gradients and higher-order derivatives. We also build a system, Teg, implementing these semantics. Our approach is widely applicable to a variety of tasks, including image stylization, fitting shader parameters, trajectory optimization, and optimizing physical designs.

Advances in Neural Rendering

“…NVIDIA OptiX is a high performance library for ray‐casting and ray‐intersection and provides to date the only possibility to use the hardware acceleration on NVIDIA RTX hardware for ray intersection. Teg [BMM*21] is a differentiable programming language which provides primitives for optimizing integrals with discontinuous integrands, as frequently found in rendering. Redner [LADL18b] is a framework for differentiable ray tracing; Mitsuba 2 [NDVZJ19] provides an even more general framework for physically based differentiable rendering and path tracing.…”

Differentiable 3D CAD Programs for Bidirectional Editing

“…Future work on optimizing through discrete changes would enable us to expand our solution to include arbitrary topological changes. Existing methods are beginning to allow differentiation through such discontinuities [BMM*21] via careful language design. One possible approach in this vein is to extend our current language to define references, user edits, and objective functions on collections of high‐level elements of variable arity rather than explicitly over a fixed number of vertices.…”

Probabilistic Programming with Programmable Variational Inference

“…Variational inference is possible in these settings, and Lee et al [36] proposed a gradient estimator that can be automated for a restricted PPL with affine discontinuities. More recently, Bangaru et al [4] and Michel et al [49] have presented techniques for differentiating integral expressions with parametric discontinuities. It is not yet clear to what extent the design we present could be cleanly extended to exploit these techniques.…”