StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Abstract

We propose StyleNeRF, a 3D-aware generative model for photo-realistic highresolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize highresolution images with fine details or yield noticeable 3D-inconsistent artifacts. In addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss. With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging tasks, including zoom-in and-out, style mixing, inversion, and semantic editing. 1

3D-Aware Indoor Scene Synthesis with Depth Priors

“…GIRAFFE [26] proposes compositional neural radiance for image rendering and accelerates the rendering process with ConvNets. StyleNeRF [11] proposes the NeRF path regularization to enforce the output to be closer to the output of NeRF that has multi-view consistency. RGBD-GAN [27] samples two camera parameters to synthesize RGBD images from two views and then warps them to each other to ensure 3D consistency.…”

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

“…To achieve this goal, the literature mainly follows two directions. The first line of works [21,34,44,46,72] utilize 3D-aware features to represent a scene, and apply a neural renderer, typically a CNN, on top of them for realistic image synthesis. For example, HoloGAN [44] and BlockGAN [45] learn lowresolution voxel features for objects, project them onto 2D image plane, and apply a StyleGAN-like [28] CNN to gen- erate higher-resolution images.…”

“…Giraffe [46] and GANcraft [22] instead use 3D volumetric rendering to generate 2D feature maps for the subsequent image generation. Following a similar idea, some works concurrent to ours [21,72] focus on designing better rendering networks to enable 3D-aware image generation at very high resolution. Nevertheless, an inevitable problem of these methods is the sacrifice of exact multiview consistency due to the learned black-box rendering.…”

VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting

“…Supervised Image Decomposition. Though producing impressive rendering quality, the geometry from 3D-aware generators is often incomplete or inaccurate [11,21]. As a result, the model tends to bias the image quality for highly sampled camera views (e.g.…”