No abstract

This paper proposes a novel algorithm to reconstruct a 3D surface from a calibrated set of images. In a first pass, it uses Scale Invariant Features Transform (SIFT) descriptor correspondences to drive the deformation of a mesh toward the true object surface. We introduce a method to handle the fact that these local descriptors are computed at positions that are not projections of mesh vertices in the images. In order to avoid projective deformations due to the large windows of interest of this descriptor, correspondences are only computed between images from the same viewpoint. This is used in a first pass to recover large concavities of the object. In a second pass, a one dimensional Lucas-Kanade tracker is used to recover small scale details. Using publicly available benchmarks, our algorithm obtains high accuracy while being among the fastest ones.

Reconstructing object geometry and material from multiple views typically requires optimization. Differentiable path tracing is an appealing framework as it can reproduce complex appearance effects. However, it is difficult to use due to high computational cost. In this paper, we explore how to use differentiable ray tracing to refine an initial coarse mesh and per-mesh-facet material representation. In simulation, we find that it is possible to reconstruct fine geometric and material detail from low resolution input views, allowing high-quality reconstructions in a few hours despite the expense of path tracing. The reconstructions successfully disambiguate shading, shadow, and global illumination effects such as diffuse interreflection from material properties. We demonstrate the impact of different geometry initializations, including space carving, multi-view stereo, and 3D neural networks. Finally, with input captured using smartphone video and a consumer 360 • camera for lighting estimation, we also show how to refine initial reconstructions of real-world objects in unconstrained environments.