Performance capture from sparse multi-view video

Abstract

Figure 1: A sequence of poses captured from eight video recordings of a capoeira turn kick. Our algorithm delivers spatio-temporally coherent geometry of the moving performer that captures both the time-varying surface detail as well as details in his motion very faithfully. AbstractThis paper proposes a new marker-less approach to capturing human performances from multi-view video. Our algorithm can jointly reconstruct spatio-temporally coherent geometry, motion and textural surface appearance of actors that perform complex and rapid moves. Furthermore, since our algorithm is purely meshbased and makes as few as possible prior assumptions about the type of subject being tracked, it can even capture performances of people wearing wide apparel, such as a dancer wearing a skirt. To serve this purpose our method efficiently and effectively combines the power of surface-and volume-based shape deformation techniques with a new mesh-based analysis-through-synthesis framework. This framework extracts motion constraints from video and makes the laser-scan of the tracked subject mimic the recorded performance. Also small-scale time-varying shape detail is recovered by applying model-guided multi-view stereo to refine the model surface. Our method delivers captured performance data at high level of detail, is highly versatile, and is applicable to many complex types of scenes that could not be handled by alternative marker-based or marker-free recording techniques.

Non-rigid Registration Meets Surface Reconstruction

“…Laplacian deformation is one the most popular techniques in this category [31] that is widely used in motion capture applications, e.g. [1]. Local geometric properties (δ-coordinates) extracted at each vertex are assumed to be preserved during the transformation.…”

“…After finding this vector, the closest rigid parameters must be found; so, the SVD decomposition is applied on the covariance matrix between the current points T(S) and updated positionT(S) 1 . Then, every affine update K i ω i can be approximated by the proper rigid parameters (R i ,t i ) to update the patch parameters:…”

Towards Plenoptic Raumzeit Reconstruction

“…Given the input images a 3D representation for each discrete time step is reconstructed and used for depth guided resampling of the plenoptic function [17][18][19][20][21][22]. For restricted scene setups the incorporation of template models proves beneficial [23][24][25]. Only few approaches reconstruct a temporally consistent mesh, which allows for continuous time interpolation [26,27].…”

Global Non-rigid Alignment of Surface Sequences

“…Research has focused on the problem of extracting a temporally consistent mesh representations over sequences by non-rigid surface tracking using either modelbased (de Aguiar et al 2008;Starck and Hilton 2003;Vlasic et al 2008) or model-free Bronstein et al 2007;Cagniart et al 2010b;Starck and Hilton 2007a;Tevs et al 2011;Tung and Matsuyama 2010;Zeng et al 2010) approaches. These single sequence alignment approaches commonly employ a sequential frame-toframe registration assuming relatively small non-rigid deformation between consecutive frames.…”