PANDORA: Polarization-Aided Neural Decomposition Of Radiance

Abstract

Reconstructing an object's geometry and appearance from multiple images, also known as inverse rendering, is a fundamental problem in computer graphics and vision. Inverse rendering is inherently ill-posed because the captured image is an intricate function of unknown lighting conditions, material properties and scene geometry. Recent progress in representing scene properties as coordinate-based neural networks have facilitated neural inverse rendering resulting in impressive geometry reconstruction and novel-view synthesis. Our key insight is that polarization is a useful cue for neural inverse rendering as polarization strongly depends on surface normals and is distinct for diffuse and specular reflectance. With the advent of commodity, on-chip, polarization sensors, capturing polarization has become practical. Thus, we propose PANDORA, a polarimetric inverse rendering approach based on implicit neural representations. From multi-view polarization images of an object, PANDORA jointly extracts the object's 3D geometry, separates the outgoing radiance into diffuse and specular and estimates the illumination incident on the object. We show that PANDORA outperforms state-of-the-art radiance decomposition techniques. PANDORA outputs clean surface reconstructions free from texture artefacts, models strong specularities accurately and estimates illumination under practical unstructured scenarios.

Disambiguation of surface normals using single-view Mueller shape-from-polarization

“…5,22 The results of the method at high SNR are also comparable to learning based methods performed on measurements. 5,20,22 Future work includes improving the computation of surface normals from the imprecise depth estimate. Applying more constraints on the smoothness and continuity of a surface is likely to improve the accuracy of the imprecise normals.…”

“…17,18 Several multi-view or stereo methods also appear in the literature. 13,14,19,20 Some approaches use multiple source positions to obtain additional information to disambiguate the normals. 21 Learning-based approaches have also been used to recover surface normals from polarimetric measurements; some include physics-based priors or view encoding into their routine.…”

“…21 Learning-based approaches have also been used to recover surface normals from polarimetric measurements; some include physics-based priors or view encoding into their routine. 5,20,22 Baek et al used optimization methods based on imprecise surface normals obtained from structured lighting, combined with a mixed specular and diffuse model. 2 This is the only work, to the authors' knowledge, that uses Mueller polarimetry to combine specular and diffuse reflection into one model to determine surface normals.…”