Few-shot Neural Human Performance Rendering from Sparse RGBD Videos

Abstract

Recent neural rendering approaches for human activities achieve remarkable view synthesis results, but still rely on dense input views or dense training with all the capture frames, leading to deployment difficulty and inefficient training overload. However, existing advances will be ill-posed if the input is both spatially and temporally sparse. To fill this gap, in this paper we propose a few-shot neural human rendering approach (FNHR) from only sparse RGBD inputs, which exploits the temporal and spatial redundancy to generate photo-realistic free-view output of human activities. Our FNHR is trained only on the key-frames which expand the motion manifold in the input sequences. We introduce a two-branch neural blending to combine the neural point render and classical graphics texturing pipeline, which integrates reliable observations over sparse key-frames. Furthermore, we adopt a patch-based adversarial training process to make use of the local redundancy and avoids over-fitting to the key-frames, which generates fine-detailed rendering results. Extensive experiments demonstrate the effectiveness of our approach to generate high-quality free view-point results for challenging human performances under the sparse setting.

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

“…In particular, the approaches with implicit function [50,52,73] reconstruct clothed humans with fine geometry details but are restricted to only human without modeling human-object interactions. For photorealistic human performance rendering, various data representation have been explored, such as point-clouds [42,64], voxels [30], implicit representations [36,47,48,62] or hybrid neural texturing [57]. However, existing solutions rely on doom-level dense RGB sensors or are limited to human priors without considering the joint rendering of human-object interactions.…”