No abstract

Introduction. Non-Rigid Shape-from-Motion (NRSfM) is the general solution to the 3D reconstruction from multiple monocular images of deforming objects. Most previous attempts in NRSfM have been on learning a low dimensional shape basis from a set of contiguous images. NRSfM is very much related to the Shape-from-Template (SfT) problem, where shape is computed from a known 3D template and a single input image after deformation. Most SfT methods have been based on isometric deformations [1,2]. Thus applying NRSfM in isometrically constrained deformations is a natural way forward. However, there has been a gap in the literature regarding the theory behind isometric NRSfM. Many of the isometric NRSfM solutions also have practical problems. Apart from that, most of the recent works in NRSfM are based on orthographic camera models.[3] uses the orthographic camera to recover the shape's normal locally; they suffer from local two-fold ambiguities and significantly degrade for shorter focal lengths.[5] recently solved the same problem for an orthographic and perspective camera.[4] specifically addresses the case of piecewise planar surfaces; it uses the perspective camera but still has patch-wise two-fold unresolved ambiguities induced by the processing of image pairs.In the paper, we present a general framework to solve Non-Rigid Shape-from-Motion (NRSfM) with the perspective camera for isometric deformations. Isometry allows solving for complex shape deformations from a sparse set of images. First, we formulate isometric NRSfM as a system of first-order Partial Differential Equations (PDE) involving the shape's depth and normal field and an unknown template. Second, we show the system cannot be locally resolved as such. Third, we introduce the concept of infinitesimal planarity and show that it makes the system locally solvable for three or more views. Finally, we derive an analytical solution which involves convex, linear least-squares optimization only, outperforming existing work on challenging datasets.

Shape-from-Template (SfT) reconstructs the shape of a deforming surface from a single image, a 3D template and a deformation prior. For isometric deformations, this is a well-posed problem. However, previous methods which require no initialization break down when the perspective effects are small, which happens when the object is small or viewed from larger distances. That is, they do not handle all projection geometries. We propose stable SfT methods that accurately reconstruct the 3D shape for all projection geometries. We follow the existing approach of using first-order differential constraints and obtain local analytical solutions for depth and the first-order quantities: the depth-gradient or the surface normal. Previous methods use the depth solution directly to obtain the 3D shape. We prove that the depth solution is unstable when the projection geometry tends to affine, while the solution for the first-order quantities remain stable for all projection geometries. We therefore propose to solve SfT by first estimating the first-order quantities (either depth-gradient or surface normal) and integrating them to obtain shape. We validate our approach with extensive synthetic and real-world experiments and obtain significantly more accurate results compared to previous initialization-free methods. Our approach does not require any optimization, which makes it very fast.

Building on progress in feature representations for image retrieval, image-based localization has seen a surge of research interest. Image-based localization has the advantage of being inexpensive and efficient, often avoiding the use of 3D metric maps altogether. That said, the need to maintain a large number of reference images as an effective support of localization in a scene, nonetheless calls for them to be organized in a map structure of some kind.The problem of localization often arises as part of a navigation process. We are, therefore, interested in summarizing the reference images as a set of landmarks, which meet the requirements for image-based navigation. A contribution of this paper is to formulate such a set of requirements for the two sub-tasks involved: map construction and self-localization. These requirements are then exploited for compact map representation and accurate selflocalization, using the framework of a network flow problem. During this process, we formulate the map construction and self-localization problems as convex quadratic and second-order cone programs, respectively. We evaluate our methods on publicly available indoor and outdoor datasets, where they outperform existing methods significantly 1 .