Inverse Path Tracing for Joint Material and Lighting Estimation

Abstract

Inverse Path Tracing Roughness Emission Albedo Rendering Geometry & Target ViewsFigure 1: Our Inverse Path Tracing algorithm takes as input a 3D scene and up to several RGB images (left), and estimates material as well as the lighting parameters of the scene. The main contribution of our approach is the formulation of an end-to-end differentiable inverse Monte Carlo renderer which is utilized in a nested stochastic gradient descent optimization. AbstractModern computer vision algorithms have brought significant advancement to 3D geometry reconstruction. However, illumination and material reconstruction remain less studied, with current approaches assuming very simplified models for materials and illumination. We introduce Inverse Path Tracing, a novel approach to jointly estimate the material properties of objects and light sources in indoor scenes by using an invertible light transport simulation. We assume a coarse geometry scan, along with corresponding images and camera poses. The key contribution of this work is an accurate and simultaneous retrieval of light sources and physically based material properties (e.g., diffuse reflectance, specular reflectance, roughness, etc.) for the purpose of editing and re-rendering the scene under new conditions. To this end, we introduce a novel optimization method using a differentiable Monte Carlo renderer that computes derivatives with respect to the estimated unknown illumination and material properties. This enables joint optimization for physically correct light transport and material models using a tailored stochastic gradient descent.

Differentiable surface splatting for point-based geometry processing

“…Nguyen-Phuoc et al [2018] and Insafutdinov and Dosovitskiy [2018] propose neural DRs using a volumetric shape representation, but the resolution is limited in practice. and Azinović et al [2019] introduce a differentiable ray tracer to implement the differentiability of physics based rendering effects, handling e.g. camera position, lighting and texture.…”

Differentiable vector graphics rasterization for editing and learning

“…Our current implementation sorts the evaluation locations by their Z-order to create coherent access, however for large images the cost of sorting is non negligible. Finally, the boundary sampling complicates selective evaluation of the pixel colors: consider a pixel-wise loss function, we can stochastically evaluate the loss function and gradients by randomly choosing the pixel locations we want to evaluate, and this significantly speeds up the gradient descent procedure [Azinović et al 2019]. Unfortunately, boundary sampling makes selective evaluation difficult: we need to sample the intersection of the boundaries and the selected pixels' filter support area.…”

In-Situ Joint Light and Medium Estimation for Underwater Color Restoration

“…However, we use a non-parameteric representation and therefore we can generalize to non-symmetric point-lights with arbitrary patterns and colors. Also in air, Azinovic et al [4] infer light source positions in a rasterization based inverse rendering scheme, which however does not extend to volumetric effects such as attenuation and scattering. Also the general concept of analysis-throughsynthesis, or inverse rendering, for obtaining optical parameters has been proposed before in air using rasterization techniques and several simulation studies have suggested that raytracing-based solutions could be useful for volumetric parameters as well [10,34,31].…”