Learned Initializations for Optimizing Coordinate-Based Neural Representations

Tancik, Matthew; Mildenhall, Ben; Wang, Terrance; Schmidt, D.; Srinivasan, Pratul P.; Barron, Jonathan T.; Ng, Ren

doi:10.1109/cvpr46437.2021.00287

Cited by 208 publications

(142 citation statements)

References 13 publications

Supporting

Mentioning

141

Contrasting

Order By: Relevance

“…Several works speed up model training by incorporating priors learned from similar datasets. Pixel-NeRF [36], IBRNet [31], and GRF [29] condition NeRF on predicted image features while Tancik et al [26] use metalearning to find good initial weight parameters that converge quickly. We view these efforts as complementary to ours.…”

Section: Related Workmentioning

confidence: 99%

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Turki¹,

Ramanan²,

Satyanarayanan³

2021

Preprint

View full text Add to dashboard Cite

We explore how to leverage neural radiance fields (NeRFs) to build interactive 3D environments from largescale visual captures spanning buildings or even multiple city blocks collected primarily from drone data. In contrast to the single object scenes against which NeRFs have been traditionally evaluated, this setting poses multiple challenges including (1) the need to incorporate thousands of images with varying lighting conditions, all of which capture only a small subset of the scene, (2) prohibitively high model capacity and ray sampling requirements beyond what can be naively trained on a single GPU, and (3) an arbitrarily large number of possible viewpoints that make it unfeasible to precompute all relevant information beforehand (as real-time NeRF renderers typically do). To address these challenges, we begin by analyzing visibility statistics for large-scale scenes, motivating a sparse network structure where parameters are specialized to different regions of the scene. We introduce a simple geometric clustering algorithm that partitions training images (or rather pixels) into different NeRF submodules that can be trained in parallel. We evaluate our approach across scenes taken from the Quad 6k and UrbanScene3D datasets as well as against our own drone footage and show a 3x training speedup while improving PSNR by over 11% on average. We subsequently perform an empirical evaluation of recent NeRF fast renderers on top of Mega-NeRF and introduce a novel method that exploits temporal coherence. Our technique achieves a 40x speedup over conventional NeRF rendering while remaining within 0.5 db in PSNR quality, exceeding the fidelity of existing fast renderers.

show abstract

Section: Related Workmentioning

confidence: 99%

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Turki¹,

Ramanan²,

Satyanarayanan³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…2K-20K iterations are sufficient for most objects, but more detail emerges when optimizing longer. Meta-learning [53] or amortization [42] could speed up synthesis.…”

Section: Discussion and Limitationsmentioning

confidence: 99%

Zero-Shot Text-Guided Object Generation with Dream Fields

Jain¹,

Mildenhall²,

Barron³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

“…(Tancik et al, 2020;Bi et al, 2020;Pumarola et al, 2021;Hennigh et al, 2021;Wang et al, 2021b;Häni et al, 2020;Zheng et al, 2020;Peng et al, 2021;Guo et al, 2020). Tancik et al (2021) use meta-learning to obtain a good initialization for fast and effective image restoration.…”

Section: Inspiring the Design Of Algorithmmentioning

confidence: 99%

Overview frequency principle/spectral bias in deep learning

Xu¹,

Zhang²,

Luo³

2022

Preprint

View full text Add to dashboard Cite

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights into this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by one-dimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.

show abstract

Learned Initializations for Optimizing Coordinate-Based Neural Representations

Cited by 208 publications

References 13 publications

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

Zero-Shot Text-Guided Object Generation with Dream Fields

Overview frequency principle/spectral bias in deep learning

Contact Info

Product

Resources

About