NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Abstract

4D modeling of human-object interactions is critical for numerous applications. However, efficient volumetric capture and rendering of complex interaction scenarios, especially from sparse inputs, remain challenging. In this paper, we propose NeuralFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors. It marries traditional non-rigid fusion with recent neural implicit modeling and blending advances, where the captured humans and objects are layerwise disentangled. For geometry modeling, we propose a neural implicit inference scheme with non-rigid key-volume fusion, as well as a template-aid robust object tracking pipeline. Our scheme enables detailed and complete geometry generation under complex interactions and occlusions. Moreover, we introduce a layer-wise human-object texture rendering scheme, which combines volumetric and image-based rendering in both spatial and temporal domains to obtain photo-realistic results. Extensive experiments demonstrate the effectiveness and efficiency of our approach in synthesizing photo-realistic free-view results under complex human-object interactions.

Human Performance Modeling and Rendering via Neural Animated Mesh

“…In the photo-realistic novel view synthesis and 3D scene modeling domain, differentiable neural rendering based on various data proxies achieves impressive results and becomes more and more popular. Various data representations are adopted to obtain better performance and characteristics, such as point-clouds [Aliev et al 2020;Suo et al 2020;], voxels [Lombardi et al 2019], texture meshes Shysheya et al 2019;Thies et al 2019] or implicit functions [Kellnhofer et al 2021;Mildenhall et al 2020;Park et al 2019] and hybrid neural blending [Jiang et al 2022a;Sun et al 2021;Suo et al 2021]. More recently, [Li et al 2020;Park et al 2020;Pumarola et al 2021;] extend neural radiance field [Mildenhall et al 2020] into the dynamic setting.…”

“…[Hu et al 2021;Peng et al 2021a,b;] utilize the human prior SMPL [Loper et al 2015] model as an anchor and use linear blend skinning algorithm to warp the radiance field. Furthermore, [Jiang et al 2022a;Sun et al 2021] extend the dynamic neural rendering and blending into the humanobjection interaction scenarios. However, for the vast majority of approaches above, dense spatial views still are required for high fidelity novel view rendering.…”

“…[Suo et al 2021] uses the occlusion map as guidance for blending weight estimation. [Jiang et al 2022a] combine the image-based blending with the per-vertex texture to solve the occlusion problem. Comparably, our neural blending scheme with spatial-temporal information which is obtained by canonical space rendering, while compressing the memory to the extreme, we can also ensure the fidelity of our blending results.…”

“…In the coarse tracking stage, we adopt the traditional non-rigid tracking [Guo et al 2015;Jiang et al 2022a;Newcombe et al 2015;Xu et al 2019a,b; based on embedded deformation to establish the coarse correspondences. As shown in Fig.…”