SimulCap : Single-View Human Performance Capture With Cloth Simulation

Abstract

This paper proposes a new method for live freeviewpoint human performance capture with dynamic details (e.g., cloth wrinkles) using a single RGBD camera. Our main contributions are: (i) a multi-layer representation of garments and body, and (ii) a physics-based performance capture procedure. We first digitize the performer using multi-layer surface representation, which includes the undressed body surface and separate clothing meshes. For performance capture, we perform skeleton tracking, cloth simulation, and iterative depth fitting sequentially for the incoming frame. By incorporating cloth simulation into the performance capture pipeline, we can simulate plausible cloth dynamics and cloth-body interactions even in the occluded regions, which was not possible in previous capture methods. Moreover, by formulating depth fitting as a physical process, our system produces cloth tracking results consistent with the depth observation while still maintaining physical constraints. Results and evaluations show the effectiveness of our method. Our method also enables new types of applications such as cloth retargeting, freeviewpoint video rendering and animations.

GarNet: A Two-Stream Network for Fast and Accurate 3D Cloth Draping

“…3D human body/cloth reconstruction. 3D body shapes/cloth are modeled from RGB/RGBD cameras in [49,43,42,15,2,1,47,48] while garment and surface reconstruction methods from images are addressed in surface/wrinkle reconstruction from images [9,3,35]. Moreover, generative models reconstruct cloths in [25,16].…”

CNN-Based Denoising, Completion, and Prediction of Whole-Body Human-Depth Images

“…With the advent of RGB-D cameras, a number of methods have been proposed to reconstruct deformable objects from monocular depth video [7]- [16]. KinectFusion [41] showed impressive results for static scenes by tracking the camera's motion with simple model-to-frame registration.…”

“…KinectFusion [41] showed impressive results for static scenes by tracking the camera's motion with simple model-to-frame registration. This seminal work was extended to deformable and dynamic objects [7] and became the basis for other subsequent works [5], [6], [8]- [16], [42]. For robust model-to-frame tracking, the methods rely on careful and slow motions [7], multiple RGB-D cameras [5], color feature tracking [8], partial resetting of the model [6], and an additional high-frame rate RGB-D camera [42].…”

“…More recent depth video-based methods [11], [13], [14], [16], [43] assume that a deformable object is a human. BodyFusion [11] deforms the surface model along with the skeleton model of a human subject.…”

“…Methods using monocular depth video [7]- [16] rely on model-to-frame tracking to join together depth measurements acquired at different times. These methods can reconstruct…”