Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes

Russell, Chris; Yu, Rui; Agapito, Lourdes

doi:10.1007/978-3-319-10584-0_38

Cited by 88 publications

(83 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, results from this field have significantly advanced. Stronger deformations have been tackled using piecewise models (Chhatkuli et al 2014;Fayad et al 2010;Russell et al 2011;Taylor et al 2010), even combining segmentation and reconstruction under local rigidity (Russell et al 2014), or eliminating the rank dependency by means of Procustean normal distributions (Lee et al 2013). In (Garg et al 2013), a variational approach integrating a low-rank shape model with spatial smoothness allowed per-pixel dense reconstructions.…”

Section: Related Workmentioning

confidence: 99%

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

Agudo

Moreno-Noguer

2016

Int J Comput Vis

View full text Add to dashboard Cite

In this paper, we simultaneously estimate camera pose and non-rigid 3D shape from a monocular video, using a sequential solution that combines local and global representations. We model the object as an ensemble of particles, each ruled by the linear equation of the Newton's second law of motion. This dynamic model is incorporated into a bundle adjustment framework, in combination with simple regularization components that ensure temporal and spatial consistency. The resulting approach allows to sequentially estimate shape and camera poses, while progressively learning a global low-rank model of the shape that is fed back into the optimization scheme, introducing thus, global constraints. The overall combination of local (physical) and global (statistical) constraints yields a solution that is both efficient and robust to several artifacts such as noisy and missing data or sudden camera motions, without requiring any training data at all. Validation is done in a variety of real application domains, including articulated and non-rigid motion, both for continuous and discontinuous shapes. Our on-line methodology yields significantly more accurate reconstructions than competing sequential approaches, being even comparable to the more computationally demanding batch methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

Agudo

Moreno-Noguer

2016

Int J Comput Vis

View full text Add to dashboard Cite

show abstract

“…Other methods proposed for 4D alignment of surface reconstructions assume that a complete mesh of the dynamic object is available for the entire sequence [21,22,23,24,25]. Partial surface tracking methods for single view [26] and RGBD data [8,27] perform sequential alignment of the reconstructions using frame-to-frame tracking. Sequential methods suffer from drift due to accumulation of errors in alignment between successive frames and failure is observed due to large non-rigid motion.…”

Section: Related Workmentioning

confidence: 99%

4D Match Trees for Non-rigid Surface Alignment

Mustafa

Kim

Hilton

2016

Computer Vision – ECCV 2016

View full text Add to dashboard Cite

Abstract. This paper presents a method for dense 4D temporal alignment of partial reconstructions of non-rigid surfaces observed from single or multiple moving cameras of complex scenes. 4D Match Trees are introduced for robust global alignment of non-rigid shape based on the similarity between images across sequences and views. Wide-timeframe sparse correspondence between arbitrary pairs of images is established using a segmentation-based feature detector (SFD) which is demonstrated to give improved matching of non-rigid shape. Sparse SFD correspondence allows the similarity between any pair of image frames to be estimated for moving cameras and multiple views. This enables the 4D Match Tree to be constructed which minimises the observed change in non-rigid shape for global alignment across all images. Dense 4D temporal correspondence across all frames is then estimated by traversing the 4D Match tree using optical flow initialised from the sparse feature matches. The approach is evaluated on single and multiple view images sequences for alignment of partial surface reconstructions of dynamic objects in complex indoor and outdoor scenes to obtain a temporally consistent 4D representation. Comparison to previous 2D and 3D scene flow demonstrates that 4D Match Trees achieve reduced errors due to drift and improved robustness to large non-rigid deformations.

show abstract

“…In this section, we briefly review the literature on monocular 3D reconstruction of deformable surfaces, i.e., template-based methods [1][2][3][4][5][6][7][8][9][10][11] and NRSfM [14][15][16][17][18][19]. In both cases, we focus the discussion on the methods that are the most relevant to our work.…”

Section: Related Workmentioning

confidence: 99%

“…By contrast, NRSfM techniques [14][15][16][17][18][19][20] do not require knowing the shape of the object a priori. Instead, they make use of a video sequence as input, and estimate the shape of the surface in all the frames of this sequence.…”

Section: Introductionmentioning

confidence: 99%

“…To overcome the ambiguities of the resulting problem, existing methods also make use of additional priors, the most popular of which is a low-rank shape basis [15,18]. As in the template-based case, impressive results have recently been achieved by NRSfM techniques, producing dense reconstructions [21] and working on natural video sequences [14]. However, to the best of our knowledge, all existing NRSfM approaches rely on tracking feature points throughout the input video sequence.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Template-Free 3D Reconstruction of Poorly-Textured Nonrigid Surfaces

Wang

Salzmann²,

Wang

et al. 2016

Computer Vision – ECCV 2016

View full text Add to dashboard Cite

Abstract. Two main classes of approaches have been studied to perform monocular nonrigid 3D reconstruction: Template-based methods and Non-rigid Structure from Motion techniques. While the first ones have been applied to reconstruct poorly-textured surfaces, they assume the availability of a 3D shape model prior to reconstruction. By contrast, the second ones do not require such a shape template, but, instead, rely on points being tracked throughout a video sequence, and are thus illsuited to handle poorly-textured surfaces. In this paper, we introduce a template-free approach to reconstructing a poorly-textured, deformable surface. To this end, we leverage surface isometry and formulate 3D reconstruction as the joint problem of non-rigid image registration and depth estimation. Our experiments demonstrate that our approach yields much more accurate 3D reconstructions than state-of-the-art techniques.

show abstract

Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes

Cited by 88 publications

References 41 publications

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

4D Match Trees for Non-rigid Surface Alignment

Template-Free 3D Reconstruction of Poorly-Textured Nonrigid Surfaces

Contact Info

Product

Resources

About