GNeRF: GAN-based Neural Radiance Field without Posed Camera

Meng, Quan; Chen, Anpei; Luo, Haimin; Wu, Minye; Su, Hao; Xu, Lu; He, Xuming; Yu, Jingyi

doi:10.1109/iccv48922.2021.00629

Cited by 146 publications

(74 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also tested BARF's [16] coarse-to-fine adjustment method to improve our poses but found that the results were generally inferior to those provided by Pix4DMapper. SCNeRF [12], GN-eRF [20], and PixSFM [17] are all recent alternatives that merit further exploration.…”

Section: B Limitationsmentioning

confidence: 99%

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Turki¹,

Ramanan²,

Satyanarayanan³

2021

Preprint

View full text Add to dashboard Cite

We explore how to leverage neural radiance fields (NeRFs) to build interactive 3D environments from largescale visual captures spanning buildings or even multiple city blocks collected primarily from drone data. In contrast to the single object scenes against which NeRFs have been traditionally evaluated, this setting poses multiple challenges including (1) the need to incorporate thousands of images with varying lighting conditions, all of which capture only a small subset of the scene, (2) prohibitively high model capacity and ray sampling requirements beyond what can be naively trained on a single GPU, and (3) an arbitrarily large number of possible viewpoints that make it unfeasible to precompute all relevant information beforehand (as real-time NeRF renderers typically do). To address these challenges, we begin by analyzing visibility statistics for large-scale scenes, motivating a sparse network structure where parameters are specialized to different regions of the scene. We introduce a simple geometric clustering algorithm that partitions training images (or rather pixels) into different NeRF submodules that can be trained in parallel. We evaluate our approach across scenes taken from the Quad 6k and UrbanScene3D datasets as well as against our own drone footage and show a 3x training speedup while improving PSNR by over 11% on average. We subsequently perform an empirical evaluation of recent NeRF fast renderers on top of Mega-NeRF and introduce a novel method that exploits temporal coherence. Our technique achieves a 40x speedup over conventional NeRF rendering while remaining within 0.5 db in PSNR quality, exceeding the fidelity of existing fast renderers.

show abstract

Section: B Limitationsmentioning

confidence: 99%

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Turki¹,

Ramanan²,

Satyanarayanan³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…These methods can reconstruct high-quality 3D shapes and perform photo-realistic view synthesis, but they have several strong assumptions on the input data, including dense camera views, precise camera parameters, and constant lighting effects. More recently, some methods [3,13,25,26,30,35] have attempted to reduce the constraints on the input data. By appending an appearance embedding to each input image, [25] can recover 3D scenes from multi-view images with different lighting effects.…”

Section: Related Workmentioning

confidence: 99%

“…By appending an appearance embedding to each input image, [25] can recover 3D scenes from multi-view images with different lighting effects. [13,26] reconstructs neural radiance fields from very sparse views by applying a discriminator to supervise the synthesized images on novel views. Different from these methods requiring multi-view images, our approach can synthesize high-resolution images by training networks only on unstructured single-view image collections.…”

Section: Related Workmentioning

confidence: 99%

3D-aware Image Synthesis via Learning Structural and Textural Representations

Xu¹,

Peng²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Making generative models 3D-aware bridges the 2D image space and the 3D physical world yet remains challenging. Recent attempts equip a Generative Adversarial Network (GAN) with a Neural Radiance Field (NeRF), which maps 3D coordinates to pixel values, as a 3D prior. However, the implicit function in NeRF has a very local receptive field, making the generator hard to become aware of the global structure. Meanwhile, NeRF is built on volume rendering which can be too costly to produce highresolution results, increasing the optimization difficulty. To alleviate these two problems, we propose a novel framework, termed as VolumeGAN, for high-fidelity 3D-aware image synthesis, through explicitly learning a structural representation and a textural representation. We first learn a feature volume to represent the underlying structure, which is then converted to a feature field using a NeRFlike model. The feature field is further accumulated into a 2D feature map as the textural representation, followed by a neural renderer for appearance synthesis. Such a design enables independent control of the shape and the appearance. Extensive experiments on a wide range of datasets show that our approach achieves sufficiently higher image quality and better 3D control than the previous methods. Project page is https://genforce.github.io/ volumegan.

show abstract

“…Neural Representations: In 3D vision, coordinate-based neural representations [7,33,34,44] have become a popular representation for various tasks such as 3D reconstruction [1,7,13,14,33,39,42,44,45,48,51,55,57], 3D-aware generative modelling [5,9,15,16,32,37,38,41,49,64], and novel-view synthesis [2,3,12,22,24,27,31,36,40,43,52,60,61]. In contrast to traditional representations like point clouds, meshes, or voxels, this paradigm represents 3D geometry and color information in the weights of a neural network, leading to a compact representation.…”

Section: Related Workmentioning

confidence: 99%

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

Niemeyer¹,

Barron²,

Mildenhall³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural Radiance Fields (NeRF) have emerged as a powerful representation for the task of novel view synthesis due to their simplicity and state-of-the-art performance. Though NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available, its performance drops significantly when this number is reduced. We observe that the majority of artifacts in sparse input scenarios are caused by errors in the estimated scene geometry, and by divergent behavior at the start of training. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints, and annealing the ray sampling space during training. We additionally use a normalizing flow model to regularize the color of unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but in many cases also conditional models that are extensively pre-trained on large multi-view datasets.

show abstract

GNeRF: GAN-based Neural Radiance Field without Posed Camera

Cited by 146 publications

References 42 publications

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

3D-aware Image Synthesis via Learning Structural and Textural Representations

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

Contact Info

Product

Resources

About