KeypointNeRF: Generalizing Image-Based Volumetric Avatars Using Relative Spatial Encoding of Keypoints

Abstract

Image-based volumetric humans using pixel-aligned features promise generalization to unseen poses and identities. Prior work leverages global spatial encodings and multi-view geometric consistency to reduce spatial ambiguity. However, global encodings often suffer from overfitting to the distribution of the training data, and it is difficult to learn multi-view consistent reconstruction from sparse views. In this work, we investigate common issues with existing spatial encodings and propose a simple yet highly effective approach to modeling high-fidelity volumetric humans from sparse views. One of the key ideas is to encode relative spatial 3D information via sparse 3D keypoints. This approach is robust to the sparsity of viewpoints and cross-dataset domain gap. Our approach outperforms state-of-the-art methods for head reconstruction. On human body reconstruction for unseen subjects, we also achieve performance comparable to prior work that uses a parametric human body model and temporal feature aggregation. Our experiments show that a majority of errors in prior work stem from an inappropriate choice of spatial encoding and thus we suggest a new direction for high-fidelity image-based human modeling.

Learning Neural Volumetric Representations of Dynamic Humans in Minutes

“…The optimization process generally takes several hours on a modern GPU, which is time-consuming and costly to scale. Inspired by multi-view stereo matching [72], some methods [8,12,45,73,87,90,99] train a network on multi-view datasets to learn to infer radiance fields from input images. This enables them to quickly fine-tune neural representations to unseen scenes.…”

Multi-Plane Neural Radiance Fields for Novel View Synthesis

“…However, in sparse-view settings, the pre-fitted predictions suffer from misalignment errors that consequently hurt the quality of the synthesized views. Mihajlovi et al [20] utilized 3D keypoints instead of body models to avoid parametric fitting errors. L-NeRF [30] introduced a time-synchronization step that accounts for the multi-view image de-synchronization by producing a per-view body model using predicted time offsets.…”

“…Comparison with generalizable NeRF methods. Generalizable human-based NeRF methods [3,12,20,44] operate only on scenes with single humans. We choose to compare against NHP [12] after adjusting it to work on multihuman scenes by using the segmentation masks to render a separate image for each individual in the scene.…”

“…A main constraint was the inefficient per-scene optimization requirement. Recently, state-of-the-art human-based synthesis methods [3,12,20,44] merged the concepts of the human model anchors and the image features to generalize to unseen poses and human identities. However, they were only designed to operate on scenes with single human subjects.…”

Single-Image 3D Human Digitization with Shape-guided Diffusion

“…Differentiable rendering based on NeRFs [Mildenhall et al 2020] has also been applied to learn 3D human representations from images. Both person-specific models [Liu et al 2021a;Peng et al 2021b;Weng et al 2022] and generalizable models across identities [Choi et al 2022;Gao et al 2022;Hu et al 2023;Kwon et al 2021;Mihajlovic et al 2022] have been proposed, but the training requires multi-view images or videos. They are difficult to collect at scale such that the collected data covers a sufficient span of clothing types and textures.…”