Volumetric Capture of Humans With a Single RGBD Camera via Semi-Parametric Learning

Abstract

Volumetric (4D) performance capture is fundamental for AR/VR content generation. Whereas previous work in 4D performance capture has shown impressive results in studio settings, the technology is still far from being accessible to a typical consumer who, at best, might own a single RGBD sensor. Thus, in this work, we propose a method to synthesize free viewpoint renderings using a single RGBD camera.The key insight is to leverage previously seen "calibration" images of a given user to extrapolate what should be rendered in a novel viewpoint from the data available in the sensor. Given these past observations from multiple viewpoints, and the current RGBD image from a fixed view, we propose an end-to-end framework that fuses both these data sources to generate novel renderings of the performer. We demonstrate that the method can produce high fidelity images, and handle extreme changes in subject pose and camera viewpoints. We also show that the system generalizes to performers not seen in the training data. We run exhaustive experiments demonstrating the effectiveness of the proposed semi-parametric model (i.e. calibration images available to the neural network) compared to other state of the art machine learned solutions. Further, we compare the method with more traditional pipelines that employ multi-view capture. We show that our framework is able to achieve compelling results, with substantially less infrastructure than previously required.

The Rocketbox Library and the Utility of Freely Available Rigged Avatars

“…Most recent techniques or generating real-time volumetric avatars are focusing on learning-based approaches. By leveraging neural rendering techniques these novel methods have achieved unprecedented results, enabling the modeling of view-dependent effects such as specularity and also correcting for imperfect geometry (Lombardi et al, 2018(Lombardi et al, , 2019Pandey et al, 2019). Compared to traditional graphics pipelines, these new methods require less computation and also can deal with a coarser geometry, which is usually used as a proxy for rendering the final 3D model.…”

The relightables

“…Machine Learning. Very recent advances in deep learning have enabled realistic synthesis of humans [Balakrishnan et al 2018;Chan et al 2018;Ma et al 2017Ma et al , 2018Neverova et al 2018;Pandey et al 2019;Si et al 2018;Zhao et al 2017]. Zhao et al [2017] use coarse-tofine Generative Adversarial Networks (GANs) to synthesize images that are still relatively blurry.…”

“…The very recent work of Pandey et al [2019] showed how to disentangle appearance, pose, and viewpoint, generating compelling renderings from arbitrary views using just a single sensor. In Lombardi et al [2019] authors provide an elegant solution to combine mesh based rendering with neural rendering by learning a 3D volume representation from multiple RGB images.…”

“…Capturing high quality, photorealistic 3D models of humans is an active area of research in the computer vision and machine learning communities [Balakrishnan et al 2018;Collet et al 2015;Dou et al 2017;Neverova et al 2018;Pandey et al 2019;Zollhöfer et al 2014]. Indeed, digital 3D humans can be employed in a variety of applications that range from photography [Meka et al 2019;Sun et al 2019] to avatars in augmented and virtual reality [Martin-Brualla et al 2018;.…”

Virtual View Synthesis Based on Asymmetric Bidirectional DIBR for 3D Video and Free Viewpoint Video

“…Therefore, wavelet transform is helpful for 3D-image coding and compression, and improves the quality of data transmission. Pandey et al [7] propose a virtual view synthesis method using a single Red-Green-Blue-Depth (RGBD) camera. It can generate novel renderings of the performer based on the past observations from multiple viewpoints and the current RGBD image from a fixed view.…”