An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing

Abstract

Abstract-The latest developments in 3D capturing, processing, and rendering provide means to unlock novel 3D application pathways. The main elements of an integrated platform, which target tele-immersion and future 3D applications, are described in this paper, addressing the tasks of real-time capturing, robust 3D human shape/appearance reconstruction, and skeletonbased motion tracking. More specifically, initially, the details of a multiple RGB-depth (RGB-D) capturing system are given, along with a novel sensors' calibration method. A robust, fast reconstruction method from multiple RGB-D streams is then proposed, based on an enhanced variation of the volumetric Fourier transform-based method, parallelized on the Graphics Processing Unit, and accompanied with an appropriate texturemapping algorithm. On top of that, given the lack of relevant objective evaluation methods, a novel framework is proposed for the quantitative evaluation of real-time 3D reconstruction systems. Finally, a generic, multiple depth stream-based method for accurate real-time human skeleton tracking is proposed. Detailed experimental results with multi-Kinect2 data sets verify the validity of our arguments and the effectiveness of the proposed system and methodologies.

Benchmarking Open-Source Static 3D Mesh Codecs for Immersive Media Interactive Live Streaming

“…There are two main directions that can be pursued for the production of 3D media. The first one reconstructs the user out of acquired sensor data in a per frame basis and is also the direction that preliminary attempts in capturing humans in full 3D initially pursued [1], [2], [22]. Recently, another direction emerged as nonrigid registration in real-time became possible.…”

“…In this work, we focus our analysis on a practical application of mesh compression, live TI streaming. Consequently, we use a set of meshes generated by an actual TI pipeline, and more specifically by the 3D reconstruction method of [22]. Our TI dataset comprises 10 randomly selected distinct frames (i.e.…”

“…As already discussed in Section 3 in the TI streaming case, the vertices of the meshes have both integer and floating point attributes. Specifically for the case of [22], they have two integer texture identifiers and one floating point blending weight for each texture pair which are used during rendering. In our experiments, the attributes corresponding to the texture identifiers are encoded losslessly while the only attribute that is subject to distortion is the one corresponding to the blending weight.…”

A Low-Cost, Flexible and Portable Volumetric Capturing System

“…Besides the expensive and laborious production of 3D media by artists using 3D modeling and animation software, there are various ways to 3D capture the real world and digitize it. Like typical video, 3D media can be consumed either in a live [2], [3] or in an on-demand manner [1], [4], with state-of-the-art systems allowing for deformations and topology changes. Offline systems typically use a pre-defined template that will be fit on the data [5] or otherwise utilize lengthy reconstruction processes [1].…”

Frequency-Domain Joint Motion and Disparity Estimation Using Steerable Filters

“…Although active depth cameras, such as Time-Of-Flight (TOF) cameras and active IR stereo pairs, introduced new potentials in these fields and hundreds of relevant papers appeared during the last decade (e.g., [3][4][5]), the use of passive RGB cameras and passive computer vision is always an active research topic, since it holds the potential to provide richer geometric representations (due to the high resolution of passive cameras), at lower cost and without capturing limitations (e.g., only indoor environments and short capturing distances).…”