Texture-Generic Deep Shape-From-Template

Abstract

Shape-from-Template (SfT) solves the registration and 3D reconstruction of a deformable 3D object, represented by the template, from a single image. Recently, methods based on deep learning have been able to solve SfT for the wide-baseline case in real-time, clearly surpassing classical methods. However, the main limitation of current methods is the need for fine tuning of the neural models to a specific geometry and appearance represented by the template texture map. We propose the first texture-generic deep learning SfT method which adapts to new texture maps at run-time, without the need for texture specific fine tuning. We achieve this by dividing the problem into a segmentation step and a registration and reconstruction step, both solved with deep learning. We include the template texture map as one of the neural inputs in both steps, training our models to adapt to different ones. We show that our method obtains comparable or better results to previous deep learning models, which are texture specific. It works in challenging imaging conditions, including complex deformations, occlusions, motion blur and poor textures. Our implementation runs in real-time, with a low-cost GPU and CPU.

State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

“…A recent real‐time SfT approach by Fuentes‐Jimenez et al . [FJPCP*21], i.e . RRNet‐DCT, relies on deep neural networks.…”

“…Its architecture has two neural networks: A segmentation module for pixel‐based detection of the template and a registration‐reconstruction module to perform SfT. RRNet‐DCT is texture‐agnostic as it adapts to new texture maps at runtime compared to the authors' earlier texture‐specific method, DeepSfT [FJPCP*21]. Being an object‐specific method that encodes the template into the neural network weights, it is highly accurate, unlike earlier object‐generic methods such as IsMo‐GAN [SGTS19].…”

“…The template often corresponds to the first frame of the sequence though it is not always a strict requirement. The template can be used as the initial state of a physics simulator [KTE*22], to obtain 3D‐2D registration as a basis for reconstruction [CPPFJ*21], and to encode prior knowledge in neural network weights [SGTS19, FJPCP*21].…”

ϕ-SfT: Shape-from-Template with a Physics-Based Deformation Model

“…These methods minimise the 3D-2D reprojection error and impose geometric constraints such as surface inextensibility [40,45] or isometry [5,34,61]. Recent neural SfT methods [13,18,41,47] predict 3D surfaces from monocular images relying on datasets with different template states. Our φ-SfT contrasts with other SfT methods in that it uses temporal information and a differentiable physics simulator as a regulariser for high-fidelity 3D surface tracking instead of approximating the underlying physical properties via geometric constraints.…”

“…The objective of SfT is: Given a known initial 3D state (a template) of an observed deformable scene or an object, reconstruct all its 3D states observed in the entire image sequence [45]. Recent learning-based SfT methods encode prior knowledge in neural network weights [13,47]. This offers multiple advantages over a vast body of previ-Figure 1.…”