MoFaNeRF: Morphable Facial Neural Radiance Field

Zhuang, Yiyu; Zhu, Hai‐Liang; Sun, Xuejing; Cao, Xun

doi:10.48550/arxiv.2112.02308

Cited by 6 publications

(7 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 3D GANs, some methods (e.g., [Athar et al 2021;Gafni et al 2021;Hong et al 2021;Zheng et al 2022;Zhuang et al 2021]) borrow latent codes for identity, expression, etc. from 3DMM (e.g.…”

Section: Neural Face Image Editingmentioning

confidence: 99%

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

Jiang

Chen

Liu

et al. 2022

SIGGRAPH Asia 2022 Conference Papers

View full text Add to dashboard Cite

Figure 1: Our NeRFFaceEditing method allows users to intuitively edit a facial volume to manipulate its geometry and appearance guided by rendered semantic masks. Given an input sample (a), our method disentangles its geometry and appearance, and allows for one-or multi-label editing. We show a range of flexible face editing tasks that can be achieved with our unified framework: (b) changing the appearance according to a given reference sample while retaining the geometry and 3D consistency; (c) changing the appearance for different views with different reference samples while retaining the geometry; (d) editing multiple labels of the semantic mask for a certain view while keeping the appearance and 3D consistency; (e) editing both the geometry and appearance. The inputs used to control the appearance and geometry are highlighted in green and orange boxes, respectively.

show abstract

“…In 3D GANs, some methods (e.g., [Athar et al 2021;Gafni et al 2021;Hong et al 2021;Zheng et al 2022;Zhuang et al 2021]) borrow latent codes for identity, expression, etc. from 3DMM (e.g.…”

Section: Neural Face Image Editingmentioning

confidence: 99%

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

Jiang

Chen

Liu

et al. 2022

SIGGRAPH Asia 2022 Conference Papers

View full text Add to dashboard Cite

show abstract

“…To break through the limited representation ability of explicit mesh based digital human representation, many works adopt the implicit representation to improve the model capacity and visual quality [Gafni et al 2021;Jiang et al 2022;Yenamandra et al 2021;Zhuang et al 2021]. i3DMM [Yenamandra et al 2021] is the first neural implicit function based 3D morphable model of full heads.…”

Section: Related Workmentioning

confidence: 99%

“…Although they usually have a good pose control over the result, but do not support expression editing due to its generative adversarial training strategy. Generic parametric head model Zhuang et al 2021] disentangles latent space of human head as identity, expression and appearance space, and to some extent realize semantical control over head transformation. However, generic head model often ignores personalized facial details and user-specific facial muscle movements due to limited MLP capacity.…”

Section: Related Workmentioning

confidence: 99%

Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video

Gao,

Zhong,

Xiang

et al. 2022

Preprint

View full text Add to dashboard Cite

facial NeRF model with only ten to twenty minutes, and can render a photorealistic human head image in tens of miliseconds with a given expression coefficient and view direction. With this novel representation, we apply it to many tasks like facial retargeting and expression editing. Experimental results demonstrate its strong representation ability and training/inference speed. Demo videos and released code are provided in our project page: https://ustc3dv.github.io/NeRFBlendShape/ CCS Concepts: • Computing methodologies → Reconstruction.

show abstract

“…Constructing accurate face models often requires a specialized capturing studio with multi-view geometry [11,20]. Recent work [3,31,32,39,41,23,9] applies deep learning methods including neural rendering to infer accurate 3D face models (see a survey [42]). They are different from our work in that they are not for novel view synthesis of the scene with a face (e.g., they do not care the scene background).…”

Section: Related Workmentioning

confidence: 99%

Novel View Synthesis for High-fidelity Headshot Scenes

Tsutsui¹,

Mao²,

Lin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Rendering scenes with a high-quality human face from arbitrary viewpoints is a practical and useful technique for many real-world applications. Recently, Neural Radiance Fields (NeRF), a rendering technique that uses neural networks to approximate classical ray tracing, have been considered as one of the promising approaches for synthesizing novel views from a sparse set of images. We find that NeRF can render new views while maintaining geometric consistency, but it does not properly maintain skin details, such as moles and pores. These details are important particularly for faces because when we look at an image of a face, we are much more sensitive to details than when we look at other objects. On the other hand, 3D Morpable Models (3DMMs) based on traditional meshes and textures can perform well in terms of skin detail despite that it has less precise geometry and cannot cover the head and the entire scene with background. Based on these observations, we propose a method to use both NeRF and 3DMM to synthesize a high-fidelity novel view of a scene with a face. Our method learns a Generative Adversarial Network (GAN) to mix a NeRF-synthesized image and a 3DMM-rendered image and produces a photorealistic scene with a face preserving the skin details. Experiments with various real-world scenes demonstrate the effectiveness of our approach. The code will be available on https://github.com/showlab/headshot .

show abstract

MoFaNeRF: Morphable Facial Neural Radiance Field

Abstract: Figure 1. MoFaNeRF is a parametric model that can synthesize free-view images by fitting to a single image or generating from a random code. The synthesized face is morphable that can be rigged to a certain expression and be edited to a certain shape or appearance.

Cited by 6 publications

References 32 publications

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video

Novel View Synthesis for High-fidelity Headshot Scenes

Contact Info

Product

Resources

About