Monocular Real-Time Volumetric Performance Capture

Abstract

We present the first approach to volumetric performance capture and novel-view rendering at real-time speed from monocular video, eliminating the need for expensive multi-view systems or cumbersome pre-acquisition of a personalized template model. Our system reconstructs a fully textured 3D human from each frame by leveraging Pixel-Aligned Implicit Function (PIFu). While PIFu achieves high-resolution reconstruction in a memory-efficient manner, its computationally expensive inference prevents us from deploying such a system for real-time applications. To this end, we propose a novel hierarchical surface localization algorithm and a direct rendering method without explicitly extracting surface meshes. By culling unnecessary regions for evaluation in a coarse-to-fine manner, we successfully accelerate the reconstruction by two orders of magnitude from the baseline without compromising the quality. Furthermore, we introduce an Online Hard Example Mining (OHEM) technique that effectively suppresses failure modes due to the rare occurrence of challenging examples. We adaptively update the sampling probability of the training data based on the current reconstruction accuracy, which effectively alleviates reconstruction artifacts. Our experiments and evaluations demonstrate the robustness of our system to various challenging angles, illuminations, poses, and clothing styles. We also show that our approach compares favorably with the state-of-the-art monocular performance capture. Our proposed approach removes the need for multi-view studio settings and enables a consumer-accessible solution for volumetric capture.

DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

“…Moreover, SMPL [28] regression or optimization can be incorporated to generate more reliable and robust outputs as shown in [48,47]. Realtime methods can be implemented with the aid of a single depth sensor [43,44] or by innovating computation and rendering algorithms [24]. Regarding to the 3D representations used in these methods, we can split them into two categories: explicit [37,30,48] and implicit [33,34,19,20] reconstruction methods.…”

“…Benefiting from the fast improvement of deep implicit functions for 3D representations, recent methods [33,34,24] are able to recover the 3D body shape only from a single RGB image. Compared with the voxel-based [37,19,48] or mesh-based [30,1] representations, an implicit function guides the deep learning models to notice geometric details in a more efficient way.…”

“…Compared with the voxel-based [37,19,48] or mesh-based [30,1] representations, an implicit function guides the deep learning models to notice geometric details in a more efficient way. Specifically, PIFu [33,24] achieves plausible single human reconstruction using only RGB images, and PIFuHD [34] further utilizes normal maps and high resolution images to generate more detailed results.…”

Human Performance Capture from Monocular Video in the Wild

“…[23,27,38,25,15,43,26,47,31] learn to infer body pose and shape from a single image, but only consider minimally clothed human. Various methods [48,60,6,42,41,18,21,59,28,36,13] have recently been proposed to reconstruct human in clothing. BodyNet [48] and DeepHuman [60] output human shape in the form of occupancy voxel grids.…”

“…Such representation has difficulties to capture fine details due to the high memory footprint. Neural implicit functions have been introduced to replace an explicit voxel grid and have enabled high-fidelity reconstructions from single images [42,41,18,21,59,28]. A major limitation of these methods is the lack of generalization to unseen poses in the wild.…”