Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

Liu, Lingjie; Habermann, Marc; Rudnev, Viktor; Sarkar, Kripasindhu; Gu, Jiatao; Theobalt, Christian

doi:10.48550/arxiv.2106.02019

Cited by 9 publications

(8 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our method utilizes a parametric 3D mesh model [3] for articulation. While we are similar to previous and concurrent works [24,21,17,29] by sharing the same goal of modeling dynamic human body with articulated NeRF representation, our method differs them in two aspects. First, we attempts to simplify the input to monocular video input as opposed to multi-view video inputs [24,17] and relax the dependence on accurate 3D geometry input [21] a priori.…”

Section: Related Workmentioning

confidence: 79%

“…A prevalent approach for representing dynamic humans with NeRF is to rig NeRF with articulated models. Common articulation choices are 3D pose skeletons [21,29] and parametric 3D mesh models [9,24,17,25]. Our method utilizes a parametric 3D mesh model [3] for articulation.…”

Section: Related Workmentioning

confidence: 99%

“…NeRF models static objects by an implicit radiance field function with multi-layer perceptron (MLP) networks. Inspired by NeRF, recent works [25,17,24,21] attempt to model 3D humans by conditioning the radiance field on 3D poses / parametric meshes. While promising human reconstruction and view synthesis results have been achieved, these methods only focus on the modeling of conditional radiance field itself and require accurate 3D poses or meshes as a prior.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Neural Capture of Animatable 3D Human from Monocular Video

Te¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a novel paradigm of building an animatable 3D human representation from a monocular video input, such that it can be rendered in any unseen poses and views. Our method is based on a dynamic Neural Radiance Field (NeRF) rigged by a mesh-based parametric 3D human model serving as a geometry proxy. Previous methods usually rely on multi-view videos or accurate 3D geometry information as additional inputs; besides, most methods suffer from degraded quality when generalized to unseen poses. We identify that the key to generalization is a good input embedding for querying dynamic NeRF: A good input embedding should define an injective mapping in the full volumetric space, guided by surface mesh deformation under pose variation. Based on this observation, we propose to embed the input query with its relationship to local surface regions spanned by a set of geodesic nearest neighbors on mesh vertices. By including both position and relative distance information, our embedding defines a distance-preserved deformation mapping and generalizes well to unseen poses. To reduce the dependency on additional inputs, we first initialize per-frame 3D meshes using off-the-shelf tools and then propose a pipeline to jointly optimize NeRF and refine the initial mesh. Extensive experiments show our method can synthesize plausible human rendering results under unseen poses and views.

show abstract

Section: Related Workmentioning

confidence: 79%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Neural Capture of Animatable 3D Human from Monocular Video

Te¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With differentiable numerical integration of volume rendering, NeRF can be trained on only posed images. Various follow-ups extend NeRF to faster training and testing [7,11,14,37,43], pose-free [23,27], dynamic scenes [4,44] and animating avatars [12,24,35]. [46] extends NeRF with a semantic segmentation renderer and boosts performance of semantic interpretation.…”

Section: Related Workmentioning

confidence: 99%

FENeRF: Face Editing in Neural Radiance Fields

Sun¹,

Wang²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Previous portrait image generation methods roughly fall into two categories: 2D GANs and 3D-aware GANs. 2D GANs can generate high fidelity portraits but with low view consistency. 3D-aware GAN methods can maintain view consistency but their generated images are not locally editable. To overcome these limitations, we propose FENeRF, a 3D-aware generator that can produce view-consistent and locally-editable portrait images. Our method uses two decoupled latent codes to generate corresponding facial semantics and texture in a spatial-aligned 3D volume with shared geometry. Benefiting from such underlying 3D representation, FENeRF can jointly render the boundary-aligned image and semantic mask and use the semantic mask to edit the 3D volume via GAN inversion. We further show such 3D representation can be learned from widely available monocular image and semantic mask pairs. Moreover, we reveal that joint learning semantics and texture helps to generate finer geometry. Our experiments demonstrate that FENeRF outperforms state-of-theart methods in various face editing tasks.

show abstract

“…To address this, several methods [12,49] propose to learn neural implicit functions to model static clothed humans. Furthermore, several methods that learn a neural avatar for a specific outfit from watertight meshes [10,14,22,30,45,51,53] have been proposed. These methods either require complete full-body scans with accurate surface normals and registered poses [10,14,51,53] or rely on complex and intrusive multi-view setups [22,30,45].…”

mentioning

confidence: 99%