Deformable Surface 3D Reconstruction from Monocular Images

Abstract

Being able to recover the shape of 3D deformable surfaces from a single video stream would make it possible to field reconstruction systems that run on widely available hardware without requiring specialized devices. However, because many different 3D shapes can have virtually the same projection, such monocular shape recovery is inherently ambiguous.In this survey, we will review the two main classes of techniques that have proved most effective so far: The template-based methods that rely on establishing correspondences with a reference image in which the shape is already known, and non-rigid structure-from-motion techniques that exploit points tracked across the sequences to reconstruct a completely unknown shape. In both cases, we will formalize the approach, discuss its inherent ambiguities, and present the practical solutions that have been proposed to resolve them. To conclude, we will suggest directions for future research.

State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

“…Several method surveys and STARs were published over the last twelve years; some of them are outdated as of 2022. Salzmann and Fua [SF10] review methods for deformable 3D surface reconstruction, only a few of which addressed the dense case in 2010. Jensen et al [HBAD21] review Non-Rigid Structure-from-Motion (NRSfM) and introduce the sparse NRSfM 2017 challenge dataset.…”

“…Shape from Template (SfT), or template-based reconstruction, comprises monocular non-rigid 3D reconstruction methods that assume a single static shape or template is given as a prior. It has been an active research area for two decades [SLF07,SF10]. The name SfT (not to be confused with Shape from Texture) became common after 2015 due to the eponymous work of Bartoli et al [BGC * 15].…”

Organic Priors in Non-Rigid Structure from Motion

“…The problem's primary objective is to recover 3D shape of a deforming object from a given set of image key-points tracked across multiple images. As a result, it is sometimes referred as solving an inverse graphics problem [46]. An effective solution to NRSf M is of significant importance to many computer vision and geometry processing applications [10,39].…”

“…For instance, the deforming shape spans a low-rank space [13], smooth temporal shape deformation [1,6], shape or trajectory lies in the union of linear subspace [26, 29-31, 35, 54] and the local surface deformation is rigid or near rigid [14,37,45,48]. Other favored prior assumptions include smooth camera motion [28,43], a piece-wise planar deformation model [14,[32][33][34], a Gaussian shape prior distribution [51], the availability of a 3D shape template [46], and shapes across frames must align [36]. Despite that, NRSf M remains a challenging and active research problem.…”