KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

Reiser, Christian O. A.; Peng, Songyou; Liao, Yiyi; Geiger, Andreas

doi:10.48550/arxiv.2103.13744

Cited by 32 publications

(49 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the low level, we plan to further optimize the GPU backend, and accelerate the CPU counterpart, potentially with cache level optimization and code generation [29], [55]. We also plan to apply ASH to sparse convolution [43], [56] and neural rendering [57], [58], where spatially varying parameterizations are exploited. ASH accelerates a variety of 3D perception workloads.…”

Section: Discussionmentioning

confidence: 99%

ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception

Dong¹,

Lao²,

Kaess³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present ASH, a modern and high-performance framework for parallel spatial hashing on GPU. Compared to existing GPU hash map implementations, ASH achieves higher performance, supports richer functionality, and requires fewer lines of code (LoC) when used for implementing spatially varying operations from volumetric geometry reconstruction to differentiable appearance reconstruction. Unlike existing GPU hash maps, the ASH framework provides a versatile tensor interface, hiding low-level details from the users. In addition, by decoupling the internal hashing data structures and key-value data in buffers, we offer direct access to spatially varying data via indices, enabling seamless integration to modern libraries such as PyTorch. To achieve this, we 1) detach stored key-value data from the low-level hash map implementation; 2) bridge the pointer-first low level data structures to index-first high-level tensor interfaces via an index heap; 3) adapt both generic and non-generic integer-only hash map implementations as backends to operate on multi-dimensional keys. We first profile our hash map against state-of-the-art hash maps on synthetic data to show the performance gain from this architecture. We then show that ASH can consistently achieve higher performance on various large-scale 3D perception tasks with fewer LoC by showcasing several applications, including 1) point cloud voxelization, 2) dense volumetric SLAM, 3) non-rigid point cloud registration and volumetric deformation, and 4) spatially varying geometry and appearance refinement. ASH and its example applications are open sourced in Open3D (http://www.open3d.org).

show abstract

Section: Discussionmentioning

confidence: 99%

ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception

Dong¹,

Lao²,

Kaess³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The field of INRs is progressing rapidly and we will likely be able to take advantage of this progress, including hybrid representations (Martel et al, 2021), better activation functions (Ramasinghe & Lucey, 2021) and reduced memory consumption (Huang et al, 2021). There has also been a plethora of work on improving NeRF (Barron et al, 2021;Reiser et al, 2021;Yu et al, 2021;Piala & Clark, 2021), which should be directly applicable to our framework. Further, recent works have shown promise for storing compressed datasets as functions (Dupont et al, 2021a;Chen et al, 2021;Strümpler et al, 2021;Zhang et al, 2021).…”

Section: Conclusion Limitations and Future Workmentioning

confidence: 99%

From data to functa: Your data point is a function and you can treat it like one

Dupont¹,

Kim²,

Eslami³

et al. 2022

Preprint

View full text Add to dashboard Cite

It is common practice in deep learning to represent a measurement of the world on a discrete grid, e.g. a 2D grid of pixels. However, the underlying signal represented by these measurements is often continuous, e.g. the scene depicted in an image. A powerful continuous alternative is then to represent these measurements using an implicit neural representation, a neural function trained to output the appropriate measurement value for any input spatial location. In this paper, we take this idea to its next level: what would it take to perform deep learning on these functions instead, treating them as data? In this context we refer to the data as functa, and propose a framework for deep learning on functa. This view presents a number of challenges around efficient conversion from data to functa, compact representation of functa, and effectively solving downstream tasks on functa. We outline a recipe to overcome these challenges and apply it to a wide range of data modalities including images, 3D shapes, neural radiance fields (NeRF) and data on manifolds. We demonstrate that this approach has various compelling properties across data modalities, in particular on the canonical tasks of generative modeling, data imputation, novel view synthesis and classification.

show abstract

“…Although, our model is naturally faster than NeRF (3X) due to that we skip the shading in empty space. We believe future works on combining mechanisms introduced in current papers such as [42,61] with our point-based radiance representation would further benefit the neural rendering technology.…”

Section: Limitationsmentioning

confidence: 99%

Point-NeRF: Point-based Neural Radiance Fields

Xu¹,

Xu²,

Philip³

et al. 2022

Preprint

View full text Add to dashboard Cite

Figure 1. Point-NeRF uses neural 3D points to efficiently represent and render a continuous radiance volume. The point-based radiance field can be predicted via network forward inference from multi-view images. It can then be optimized per scene to achieve reconstruction quality that surpasses NeRF [35] in tens of minutes. Point-NeRF can also leverage off-the-shelf reconstruction methods like COLMAP [44]and is able to perform point pruning and growing that automatically fix the holes and outliers that are common in these approaches.

show abstract

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

Cited by 32 publications

References 48 publications

ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception

ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception

From data to functa: Your data point is a function and you can treat it like one

Point-NeRF: Point-based Neural Radiance Fields

Contact Info

Product

Resources

About