PVA: Pixel-aligned Volumetric Avatars

Raj, Amit; Zollhoefer, Michael; Simon, Tomas; Saragih, Jason; Saito, Shunsuke; Hays, James; Lombardi, Stephen

doi:10.48550/arxiv.2101.02697

Cited by 9 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Baselines. Among the recent generalizable NeRF methods [36,52,47], we compare with Pixel-NeRF [52] and PVA [36] which focus on very sparse (up to 3 or 4) input views. we reimplement [36] since it is not open-sourced.…”

Section: Comparison With Generalizable Nerf Methodsmentioning

confidence: 99%

“…Despite the promising results, these general NeRF [19,53] and human-specific NeRF [13,32,33,35,50] methods must be optimized for each new video separately, and generalize poorly on unseen scenarios. Generalizable NeRFs [36,47,52] try to avoid the expensive per-scene optimization by imageconditioning using pixel-aligned features. However, directly extending such methods to model complex and dynamic 3D humans is not straightforward when available observations are highly sparse.…”

Section: Related Workmentioning

confidence: 99%

“…First, we construct the time-augmented skeletal features {s}. We exploit a human body model (SMPL [26]) to construct 3D skeletal features by projecting all the SMPL vertices onto each memory frame and picking up the pixel-aligned image features [36,37,52] at the projected 2D locations. The skeletal features are sampled from all memory frames to construct the skeletal feature bank.…”

Section: Neural Human Performermentioning

confidence: 99%

“…Recently, neural radiance fields (NeRF) [28,13,19,32,33,35,36,47,50,52,53] have shown photo-realistic novel view synthesis results in per-scene optimization settings. To avoid the expensive per-scene training and improve the practicality, generalizable NeRFs [36,52,47] have been proposed which use image-conditioned, pixel-aligned features and achieve feed-forward view synthesis from sparse input views [36,52]. Direct application of these methods to complex and non-rigid human motion is not straightforward, however, and naive solutions suffer from significant artifacts as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

Kwon,

Kim,

Ceylan

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code are available at https://youngjoongunc.github.io/nhp.Preprint. Under review.

show abstract

Section: Comparison With Generalizable Nerf Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Neural Human Performermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

Kwon,

Kim,

Ceylan

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Classic 3D reconstruction uses SfM pipelines from multiple views of a rigid scene, with pipelines such as KinectFusion [35] and Dynam-icFusion [36] integrating depth sensors for 3D reconstruction of dense static and deformable surfaces. NeRF [34] and its deformable extensions [44,7,55,46,40] such as D-NeRF [45] synthesize novel views from densely sampled multi-views of a static or mildly dynamic scene using a Neural Radiance Field, from which explicit 3D geometry can be further extracted. A more recent work, LASR [63], optimizes a single 3D deformable model on an individual video sequence.…”

Section: Related Workmentioning

confidence: 99%

DOVE: Learning Deformable 3D Objects by Watching Videos

Wu¹,

Jakab²,

Rupprecht³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning deformable 3D objects from 2D images is an extremely ill-posed problem. Existing methods rely on explicit supervision to establish multi-view correspondences, such as template shape models and keypoint annotations, which restricts their applicability on objects "in the wild". In this paper, we propose to use monocular videos, which naturally provide correspondences across time, allowing us to learn 3D shapes of deformable object categories without explicit keypoints or template shapes. Specifically, we present DOVE, which learns to predict 3D canonical shape, deformation, viewpoint and texture from a single 2D image of a bird, given a bird video collection as well as automatically obtained silhouettes and optical flows as training data. Our method reconstructs temporally consistent 3D shape and deformation, which allows us to animate and re-render the bird from arbitrary viewpoints from a single image. IntroductionRecently, with the adoption of machine learning, reconstructing 3D shapes from 2D images has advanced considerably. While this task traditionally requires establishing correspondences between multiple views [13], learning-based approaches have demonstrated the possibility of inferring 3D shapes from a single image, by learning priors for a specific object category [

show abstract

Voxel grid performer: efficient radiance fields generalization for human novel view synthesis

Fan

Liu

Zhao

et al. 2022

Optical Metrology and Inspection for Industrial Applications IX

View full text Add to dashboard Cite

Novel view synthesis is a long-standing problem. Despite the rapid development of neural radiance field (NeRF), in terms of rendering dynamic human body, NeRF still cannot achieve a good trade-off in precision and efficiency. In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performers in an efficient way, only requiring a sparse number of camera views as inputs and skirting per-case fine-tuning. Recently, several works have addressed this problem by learning person-specific neural radiance fields to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting these generalization aapprochs to human achieve reasonable rendering result. However, due to the difficulties of modeling the complex appearance of human and the dynamic sense, it is challenging to train nerf well in an efficient way. We find that the slow convergence of the human body reconstruction model is largely due to the nerf representation. In this work, we introduce a voxel grid based representation for human view synthesis, termed Voxel Grid Performer(VGP). Specifically, a sparse voxel grid is designed to represent the density and color in every sapce voxel, which enable better performance and less computation than conventional nerf optimization. We perform extensive experiments on both seen human performer and unseen human performer, demonstrating that our approach surpasses nerf-based methods on a wide variety of metrics. Code and data will be made available at https://github.com/fanzhongyi/vgp.

show abstract

PVA: Pixel-aligned Volumetric Avatars

Abstract: Figure 1: We present a novel approach for the prediction of volumetric avatars of human heads from a small number of example views. Our model enables view synthesis for unseen identities and is able to generate faithful facial expressions.

Cited by 9 publications

References 21 publications

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

DOVE: Learning Deformable 3D Objects by Watching Videos

Voxel grid performer: efficient radiance fields generalization for human novel view synthesis

Contact Info

Product

Resources

About