3D Deformable Shape Reconstruction with Diffusion Maps

Abstract

This paper presents a method for recovering deformable shape and motion from uncalibrated 2D video sequence in the presence of missing data. Highly deformable shapes are hard to describe under previously used assumptions, such as global constraint enforcing shapes to lie within a linear subspace. Considering that the data dimensionality may not represent the true complexity of the problem, we suggest that the shapes can be well-modelled in a low dimensional manifold. However, building a dense representation of the manifold requires a large amount of training data which is not feasible in many real applications. The main contribution of this paper is to propose a novel approach for estimating accurate 3D reconstructions utilising manifold learned from a relatively small number of training samples. The problem is addressed by grouping shapes into evolving clusters, with the shapes in each cluster represented in the linear subspace, estimated based on the observations and the prior learned manifold. Results are presented using motion capture data and real video sequences, showing that the proposed method can better model shapes with complex deformations compare to several state-of-the-art techniques, and is robust against noise and missing data.

Top-down model fitting for hand pose recovery in sequences of depth images

“…Temporal information and trajectory analysis, besides the shape itself, provide discriminative information to analyze shape and recover occluded parts. Works from structure from motion, such as matrix imputation [23], statistical model analysis and non-rigid structure from motion [16,29], showed the benefits of using temporal information for shape analysis. Zhou et al [37] proposed a spatio-temporal model for the problem of human pose recovery.…”

“…Time-varying spatial data is involved in a vast range of computer vision applications [33,29] and proved to be useful in extracting missing data. Spatial correlation or trajectory analysis of independent points solely fails to model all information in spatio-temporal data.…”

Robust deformable shape reconstruction from monocular video with manifold forests

“…The similar method was introduced in [32], but manifold was learned without using the random forest.…”

“…This paper updates and extends the work in [33] with the following four main differences: (a) The method presented in this paper has an additional step in the algorithm, solving the problem when some elements of the measurement matrix are missing; (b) Considering the majority of algorithms are based on minimising squared residual of an error function which makes them sensitive to outliers, another improvement is to reduce the effect of outliers by replacing the L 2 estimator by robust M-estimator [26]; (c) A modification of the method is described when only a relatively small number of training shapes is available. This was firstly introduced by the authors in [32] but without random forests manifold learning technique; (d) More comprehensive set of experiments is presented in the experimental section.…”