Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity

Abstract

Introduction. Non-Rigid Shape-from-Motion (NRSfM) is the general solution to the 3D reconstruction from multiple monocular images of deforming objects. Most previous attempts in NRSfM have been on learning a low dimensional shape basis from a set of contiguous images. NRSfM is very much related to the Shape-from-Template (SfT) problem, where shape is computed from a known 3D template and a single input image after deformation. Most SfT methods have been based on isometric deformations [1,2]. Thus applying NRSfM in isometrically constrained deformations is a natural way forward. However, there has been a gap in the literature regarding the theory behind isometric NRSfM. Many of the isometric NRSfM solutions also have practical problems. Apart from that, most of the recent works in NRSfM are based on orthographic camera models.[3] uses the orthographic camera to recover the shape's normal locally; they suffer from local two-fold ambiguities and significantly degrade for shorter focal lengths.[5] recently solved the same problem for an orthographic and perspective camera.[4] specifically addresses the case of piecewise planar surfaces; it uses the perspective camera but still has patch-wise two-fold unresolved ambiguities induced by the processing of image pairs.In the paper, we present a general framework to solve Non-Rigid Shape-from-Motion (NRSfM) with the perspective camera for isometric deformations. Isometry allows solving for complex shape deformations from a sparse set of images. First, we formulate isometric NRSfM as a system of first-order Partial Differential Equations (PDE) involving the shape's depth and normal field and an unknown template. Second, we show the system cannot be locally resolved as such. Third, we introduce the concept of infinitesimal planarity and show that it makes the system locally solvable for three or more views. Finally, we derive an analytical solution which involves convex, linear least-squares optimization only, outperforming existing work on challenging datasets.

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

“…(8) using sparse Levenberg-Marquardt. Note, again, that in contrast to competing approaches (Bregler et al 2000;Dai et al 2012), we can deal with missing data and do not require all points to be tracked throughout the whole sequence.…”

“…On top of this, spatial (Torresani et al 2008) and temporal (Bartoli et al 2008;Del Bue et al 2006;Torresani et al 2008) smoothness priors have been considered to further limit the solution space. Later, Dai et al (2012) relaxed the amount of extra prior knowledge by directly imposing a low-rank constraint on the factorization of the measurement matrix. Other approaches have modeled deformation using a low-rank trajectory basis per 3D point (Akhter et al 2008), including priors on trajectories in terms of 3D point differentials (Valmadre and Lucey 2012) and enforcing smoothness on their paths (Gotardo and Martínez 2011b).…”

“…Recently, results from this field have significantly advanced. Stronger deformations have been tackled using piecewise models (Chhatkuli et al 2014;Fayad et al 2010;Russell et al 2011;Taylor et al 2010), even combining segmentation and reconstruction under local rigidity (Russell et al 2014), or eliminating the rank dependency by means of Procustean normal distributions (Lee et al 2013). In (Garg et al 2013), a variational approach integrating a low-rank shape model with spatial smoothness allowed per-pixel dense reconstructions.…”

“…This is in contrast to most existing NRSfM approaches, that typically apply constraints in a global manner, usually in the form of low-rank models. For batch methods (Dai et al 2012;Del Bue et al 2006;Garg et al 2013;Torresani et al 2008), these low-rank models are learned after processing all frames of the sequence at once. For sequential ones (Paladini et al 2010;Tao et al 2013), the lowrank model is incrementally learned.…”

“…Yet, while low-rank meth-ods can effectively encode global deformations, they cannot, in general, handle non-linear motion patterns and strong local deformations. Piecewise strategies (Chhatkuli et al 2014;Russell et al 2011;Taylor et al 2010;Varol et al 2009) allow recovering larger deformations, although their performance highly depends on having overlapping features in neighboring patches, or require large number of correspondences to enforce local rigidity constraints (Chhatkuli et al 2014;Taylor et al 2010;Varol et al 2009), which can be hard to obtain in practice. In any event, these previous approaches batch process all frames of the sequence at once, after video capture, preventing them from being used online and in real-time applications, where NRSfM may have an enormous potential.…”