Accurate 3D Reconstruction of Dynamic Scenes from Monocular Image Sequences with Severe Occlusions

Golyanik, Vladislav; Fetzer, Torben; Stricker, Didier

doi:10.1109/wacv.2017.38

Cited by 14 publications

(25 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, some of the methods perform a low-rank approximation of a stacked shape matrix [18,26]. A further milestone is associated with the ability to perform dense reconstructions [3,6,26,31,33].…”

Section: Non-rigid Structure From Motionmentioning

confidence: 99%

“…All approaches to MNR can be divided into two main model-based classesnon-rigid structure from motion (NRSfM) and template-based reconstruction (TBR). NRSfM relies on motion and deformation cues and requires dense point correspondences over multiple frames [26,31]. Most accurate methods for dense correspondences operate on multiple frames and are prohibitively slow for real-time applications [66].…”

Section: Introductionmentioning

confidence: 99%

“…In some cases, it relies on a known camera trajectory or proxy geometry (an initial coarse geometry estimate) [7]. In hybrid NRSfM, a scene-specific shape prior is obtained on-the-fly under non-rigidity, and the input is a sequence of point tracks [31]. Geometry estimation is then conditioned upon the shape prior.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

HDM-Net: Monocular Non-rigid 3D Reconstruction with Learned Deformation Model

Golyanik

Shimada

Varanasi

et al. 2018

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call Hybrid Deformation Model Network (HDM-Net). In our approach, a deformation model is learned by a deep neural network, with a combination of domainspecific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, shading and contours. We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under new illumination conditions. Experiments with noisy data and a comparison with other methods demonstrate the robustness and accuracy of the proposed approach and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality.

show abstract

Section: Non-rigid Structure From Motionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HDM-Net: Monocular Non-rigid 3D Reconstruction with Learned Deformation Model

Golyanik

Shimada

Varanasi

et al. 2018

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…If no template is available, a shape prior can be estimated on-the-fly from dense point correspondences obtained on several unoccluded views. The shape prior can then be used as a constraint for reconstruction of the occluded areas, both with and without available correspondences for the rest of the sequence [19]. Finally, correspondence based methods or non-rigid structure from motion (NRSfM) 1 can employ correspondence correction in the preprocessing step [36], although this approach works well for rather short-time disturbances.…”

Section: Introductionmentioning

confidence: 99%

“…We elaborate efficient optimisation techniques involving direct and inverse fast Fourier transforms (FT) and show experimentally that the proposed form of regularisation has several advantages (e.g., ability to filter depth values flexibly without edge oversmoothing) both when dealing with occluded and unoccluded scenes. CDF achieves state of the art accuracy on the joint evaluation benchmark with large occlusions [19], an actor benchmark [5] and several established data sets for qualitative evaluation (with and without occlusions).…”

mentioning

confidence: 99%

Introduction to Coherent Depth Fields for Dense Monocular Surface Recovery

Golyanik¹,

Fetzer²,

Stricker³

2017

Procedings of the British Machine Vision Conference 2017

Self Cite

View full text Add to dashboard Cite

Handling large occlusions in non-rigid structure from motion (NRSfM) currently requires either an expensive correspondence correction or estimation of a shape prior on several non-occluded views. To save computational cost and remove the dependency on additional pre-processing steps, this paper introduces the concept of depth fields. With the proposed depth fields, NRSfM is interpreted as an alternating estimation of vector fields with fixed origins on the one side, and estimation of displacements of the origins along the depth dimension on the other. The core of the new energy-based Coherent Depth Fields (CDF) approach is the spatial smoothness coherency term (CT) applied on the depth fields. Having its origins in the Motion Coherence Theory, CT interprets data as a displacement vector field and penalises irregularities in displacements. Not only for handling occlusions but also for unoccluded scenes CT has multiple advantages compared to previously proposed regularisers such as total variation. We show experimentally that CDF achieves state-of-the-art in dense NRSfM including scenarios with long and large occlusions, inaccurate correspondences as well as inaccurate initialisations, without requiring any additional pre-processing steps. c 2017. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.Contributions. This paper introduces two novel concepts which allow to design a computationally efficient, accurate and easy to implement (practical) approach as well as overcome the dependency on the pre-processing steps in NRSfM while handling large occlusions. The first concept is the notion of a depth vector field or, concisely, a depth field. A depth field is a 2D parametrisation of a surface embedded into 3D space so that every tracked 2D point is associated with a displacement along the depth dimension. This definition implies that all displacements are parallel to each other, or, in other words, a depth field is an irrotational vector field. With the new parametrisation the problem of NRSfM is interpreted as a filtering of a depth field in the first alternating step (at that moment point origins are fixed), and shape refinement in the second alternation step. The updated shape, in turn, alters the depth field which is further filtered, and so on until convergence.Next, we propose coherency term (CT) as a new soft spatial regulariser on the adjacent depth vectors. CT derives its origin from the motion coherence theory (MCT) [43,44] which studies principles of coherent motion and perception. MCT, in accordance to the human visual system states that neighboring structures tend to move coherently, i.e., with a common velocity and direction. We call the proposed approach Coherent Depth Fields (CDF) and formulate it as an energy-minimisation problem with CT. The main reason lies in the expressiveness of energy-based methods -an energy functional explicitly encodes assumptions on the underlying physical processes and relates input data with the sou...

show abstract

Self-Calibrating Isometric Non-Rigid Structure-from-Motion

Parashar

Bartoli

Pizarro

2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

Non-Rigid Structure-from-Motion (NRSfM) reconstructs a deformable 3D object from the correspondences established between monocular 2D images. Current NRSfM methods lack statistical robustness, which is the ability to cope with correspondence errors. This prevents one to use automatically established correspondences, which are prone to errors, thereby strongly limiting the scope of NRSfM. We propose a three-step automatic pipeline to solve NRSfM robustly by exploiting isometry. Step (i) computes the optical flow from correspondences, step (ii) reconstructs each 3D point's normal vector using multiple reference images and integrates them to form surfaces with the best reference and step (iii) rejects the 3D points that break isometry in their local neighborhood. Importantly, each step is designed to discard or flag erroneous correspondences. Our contributions include the robustification of optical flow by warp estimation, new fast analytic solutions to local normal reconstruction and their robustification, and a new scale-independent measure of 3D local isometric coherence. Experimental results show that our robust NRSfM method consistently outperforms existing methods on both synthetic and real datasets.

show abstract

Accurate 3D Reconstruction of Dynamic Scenes from Monocular Image Sequences with Severe Occlusions

Cited by 14 publications

References 51 publications

HDM-Net: Monocular Non-rigid 3D Reconstruction with Learned Deformation Model

HDM-Net: Monocular Non-rigid 3D Reconstruction with Learned Deformation Model

Introduction to Coherent Depth Fields for Dense Monocular Surface Recovery

Self-Calibrating Isometric Non-Rigid Structure-from-Motion

Contact Info

Product

Resources

About