UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

Abstract

Neural implicit 3D representations have emerged as a powerful paradigm for reconstructing surfaces from multiview images and synthesizing novel views. Unfortunately, existing methods such as DVR or IDR require accurate perpixel object masks as supervision. At the same time, neural radiance fields have revolutionized novel view synthesis. However, NeRF's estimated volume density does not admit accurate surface reconstruction. Our key insight is that implicit surface models and radiance fields can be formulated in a unified way, enabling both surface and volume rendering using the same model. This unified perspective enables novel, more efficient sampling procedures and the ability to reconstruct accurate surfaces without input masks. We compare our method on the DTU, BlendedMVS, and a synthetic indoor dataset. Our experiments demonstrate that we outperform NeRF in terms of reconstruction quality while performing on par with IDR without requiring masks.

Volume Rendering of Neural Implicit Surfaces

“…Independently from and concurrently with our work here, [30] also use implicit surface representation incorporated into volume rendering. In particular, they replace the local transparency function with an occupancy network [22].…”

Deep Medial Fields

“…For neural implicit shape representations, differentiable renderers have been proposed to learn implicit representations of geometry given only 2D observations of 3D scenes Sitzmann et al 2019;]. Alternatively, one may parameterize density and radiance of a 3D scene, enabling volumetric rendering , or combine volumetric and ray-marching based approaches [Oechsle et al 2021]. As rendering of neural implicit representations requires hundreds of evaluations of the distance field per ray, hybrid explicit-implicit representations have been proposed to provide significant speedups [Takikawa et al 2021].…”

“…As such, the SDF value can be used to bound the step size of a ray-marching algorithm in a way that guarantees that overstepping will not occur. In many cases, such as the case of a ray pointed directly orthogonal to a flat surface, sphere tracing will converge in one or very few iterations, making it an attractive option for rendering which is frequently used in applications that handle implicit surfaces [Oechsle et al 2021;Seyb et al 2019;Sitzmann et al 2019;Takikawa et al 2021;]. However, there are pathological cases in which sphere tracing may take arbitrarily many iterations for a ray to converge.…”

Representing 3D Shapes with Probabilistic Directed Distance Fields

“…In comparison to explicit representations, implicit shapes can capture arbitrary topologies with high fidelity. Several works examine differentiable rendering of implicit fields [24,37,38,48,67,69] (or combine it with neural volume rendering [27,50,77,81]). In contrast, by conditioning on both viewpoint and position, DDFs can flexibly render depth, with a single field query per pixel.…”

“…However, the standard differentiable volume render- ing formulation of NeRFs is computationally expensive, requiring many forward passes per pixel, though recent work has improved on this (e.g., [2,10,17,27,35,56,57,82]). Furthermore, the distributed nature of the density makes extracting explicit geometric details (including higher-order surface information) more difficult (e.g., [50,81]).…”