VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting

Tan, Feitong; Fanello, Sean; Meka, Abhimitra; Orts-Escolano, Sergio; Tang, Dajun; Pandey, Rohit; Taylor, Jonathan; Tan, P.; Zhang, Yinda

doi:10.48550/arxiv.2201.04873

Cited by 3 publications

(3 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, their estimated geometry is still too smooth and lacks the precision for conducting other stylization tasks such as relighting. Most recently, [Tan et al 2022] imposes additional lighting constraints to simultaneously achieve relighting and perspective changes. Yet, the final quality still falls short of the photographic level due to volumetric neural rendering which also causes cross-view flickering .…”

Section: Related Workmentioning

confidence: 99%

NARRATE: A Normal Assisted Free-View Portrait Stylizer

Wang¹,

Xu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

face model to render more detailed images. In particular, we fuse the neural and physical renderings to compensate for the imperfect inversion, resulting in both realistic and view-consistent novel perspective images.In relighting stage, previous works Sun et al. 2019] focus on single view portrait relighting but ignoring consistency between different perspectives as well, leading unstable and inconsistent lighting effects for view changes. We extend to fix this problem by unifying its multi-view input normal maps with the physical face model. NARRATE conducts relighting with consistent normal maps, imposing crossview constraints and exhibiting stable and coherent illumination effects.We experimentally demonstrate that NARRATE achieves more photorealistic, reliable results over prior works. We further bridge NARRATE with animation and style transfer tools, supporting pose change, light change, facial animation, and style transfer, either separately or in combination, all at a photographic quality. We showcase vivid free-view facial animations as well as 3D-aware relightable stylization, which help facilitate various AR/VR applications like virtual cinematography, 3D video conferencing, and post-production.

show abstract

Section: Related Workmentioning

confidence: 99%

NARRATE: A Normal Assisted Free-View Portrait Stylizer

Wang¹,

Xu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other Face Relighting Methods mostly use carefully collected supervisory data from lightstages [41,57,28,36,30]. ShadeGAN [29] and Volux-GAN [42] uses a volumetric rendering approach to learn the underlying 3D structure of the face and the illumination encoding. Volux-GAN also requires image decomposition obtained from [30] that is trained using a carefully curated light-stage data.…”

Section: Related Workmentioning

confidence: 99%

Enriching StyleGAN with Illumination Physics

Bhattad¹,

Forsyth²

2022

Preprint

View full text Add to dashboard Cite

Left column shows images generated by a vanilla StyleGAN2, kindly provided by [54].Other columns show relights of the scenes in these images. We obtain these images by adding directions to the latent style code coefficients (w + ) of the original images. Directions are independent of w + , and are obtained by a search for directions that produce diverse changes in images without changing albedo. Note extreme changes in illumination of the scene; the geometry and albedo of the scenes do not change; and the images are realistic.

show abstract

“…People address these scaling issues of NeRF-based GANs in different ways, but the dominating approach is to train a separate 2D decoder to produce a high-resolution image from a low-resolution image or feature grid rendered from a NeRF backbone [43]. During the past six months, there appeared more than a dozen of methods which follow this paradigm (e.g., [6,15,71,47,79,35,75,23,72,78,64]). While using the upsampler allows to scale the model to high resolution, it comes with two severe limitations: 1) it breaks multi-view consistency of a generated object, i.e.…”

Section: Introductionmentioning

confidence: 99%

EpiGRAF: Rethinking training of 3D GANs

Skorokhodov¹,

Tulyakov²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

A very recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. During the past months, there appeared more than 10 works that address this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator. But this solution comes at a cost: not only does it break multi-view consistency (i.e. shape and texture change when the camera moves), but it also learns the geometry in a low fidelity. In this work, we show that it is possible to obtain a high-resolution 3D generator with SotA image quality by following a completely different route of simply training the model patch-wise. We revisit and improve this optimization scheme in two ways. First, we design a location-and scale-aware discriminator to work on patches of different proportions and spatial positions. Second, we modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. The resulted model, named EpiGRAF, is an efficient, high-resolution, pure 3D generator, and we test it on four datasets (two introduced in this work) at 256 2 and 512 2 resolutions. It obtains state-of-the-art image quality, high-fidelity geometry and trains ≈2.5× faster than the upsampler-based counterparts.

show abstract

VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting

Abstract: Figure 1. We propose VoLux-GAN, a 3D-aware generator that produces faces with full HDRI relighting capability. Here we show a comparison of images generated by VoLux-GAN and related work pi-GAN [11] (which does not support relighting) and ShadeGAN [43].

Cited by 3 publications

References 46 publications

NARRATE: A Normal Assisted Free-View Portrait Stylizer

NARRATE: A Normal Assisted Free-View Portrait Stylizer

Enriching StyleGAN with Illumination Physics

EpiGRAF: Rethinking training of 3D GANs

Contact Info

Product

Resources

About