Deep Textured 3D Reconstruction of Human Bodies

Abstract

Recovering textured 3D models of non-rigid human body shapes is challenging due to self-occlusions caused by complex body poses and shapes, clothing obstructions, lack of surface texture, background clutter, sparse set of cameras with non-overlapping fields of view, etc. Further, a calibration-free environment adds additional complexity to both -reconstruction and texture recovery. In this paper, we propose a deep learning based solution for textured 3D reconstruction of human body shapes from a single view RGB image. This is achieved by first recovering the volumetric grid of the non-rigid human body given a single view RGB image followed by orthographic texture view synthesis using the respective depth projection of the reconstructed (volumetric) shape and input RGB image. We propose to co-learn the depth information readily available with affordable RGBD sensors (e.g., Kinect) while showing multiple views of the same object during the training phase. We show superior reconstruction performance in terms of quantitative and qualitative results, on both, publicly available datasets (by simulating the depth channel with virtual Kinect) as well as real RGBD data collected with our calibrated multi Kinect setup.

Single-View 3D Reconstruction via Differentiable Rendering and Inverse Procedural Modeling

“…There are a number of methods focused on the reconstruction of objects with geometric constraints, such as bodies of revolution [53,54], buildings [55], human bodies [56,57], etc. These approaches are carefully designed for specific application scenarios and specific input data.…”

Taxonomy and Survey of Current 3D Photorealistic Human Body Modelling and Reconstruction Techniques for Holographic-Type Communication

“…For the purposes of 3D human body reconstruction, some approaches [58,[63][64][65]92] exploit depth maps that are generated by specific systems. These kinds of systems use structured light or the Time of Flight principle to measure the depth of an object, which shows how far from the system the object is.…”

HumanMeshNet: Polygonal Mesh Recovery of Humans

“…However, HMNetOracle produces a significant increase in accuracy with the increase in quality of the input image and segmentation mask (Table 4). Similar to state of the art methods [28,30,12], we rely on 3D body supervision and providing more supervision like sillhoute and 2D keypoint loss like [28,11] can improve the performance further. For Human3.6m, we compare with those that don't use mesh supervision (since this data is currently unavailable) and achieve comparable performance.…”

“…Recovering a 3D human body shape from a monocular image is an ill posed problem in computer vision with great practical importance for many applications, including virtual and augmented reality platforms, animation industry, e-commerce domain, etc. Some of the recent deep Figure 1: We present an early method to integrate Deep Learning with the sparse mesh representation, to successfully reconstruct the 3D mesh of a human from a monocular image learning methods employ volumetric regression to recover the voxel grid reconstruction of human body models from a monocular image [30,28]. Although volumetric regression enables recovering a more accurate surface reconstruction, they do so without an animatable skeleton [30], which limits their applicability for some of the aforementioned applications.…”

“…Some of the recent deep Figure 1: We present an early method to integrate Deep Learning with the sparse mesh representation, to successfully reconstruct the 3D mesh of a human from a monocular image learning methods employ volumetric regression to recover the voxel grid reconstruction of human body models from a monocular image [30,28]. Although volumetric regression enables recovering a more accurate surface reconstruction, they do so without an animatable skeleton [30], which limits their applicability for some of the aforementioned applications. [28] attempted to overcome this limitation by fitting a parametric body model on the volumetric reconstruction using a sillhoute reprojection loss.…”