VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field

Abstract

Neural Radiance Field (NeRF) is a popular method in data-driven 3D reconstruction. Given its simplicity and high quality rendering, many NeRF applications are being developed. However, NeRF's big limitation is its slow speed. Many attempts are made to speeding up NeRF training and inference, including intricate code-level optimization and caching, use of sophisticated data structures, and amortization through multi-task and meta learning. In this work, we revisit the basic building blocks of NeRF through the lens of classic techniques before NeRF. We propose Voxel-Accelearated NeRF (VaxNeRF), integrating NeRF with visual hull, a classic 3D reconstruction technique only requiring binary foreground-background pixel labels per image. Visual hull, which can be optimized in about 10 seconds, can provide coarse in-out field separation to omit substantial amounts of network evaluations in NeRF. We provide a clean fully-pythonic, JAX-based implementation on the popular JaxNeRF codebase, consisting of only about 30 lines of code changes and a modular visual hull subroutine, and achieve about 2-8x faster learning on top of the highly-performative JaxNeRF baseline with zero degradation in rendering quality. With sufficient compute, this effectively brings down full NeRF training from hours to 30 minutes. We hope VaxNeRF -a careful combination of a classic technique with a deep method (that arguably replaced it) -can empower and accelerate new NeRF extensions and applications, with its simplicity, portability, and reliable performance gains. Codes are available at https://github.com/naruya/VaxNeRF. (a) NeRF (b) VaxNeRF Figure 1. NeRF (a) uses hierarchical sampling, i.e., a mixture of coarse (pink) and fine (light blue) sampling, to estimate the color from a specified viewpoint. Instead, VaxNeRF (b) samples only the inside of the bounding volume uniformly and densely. The orange circle shows the area bounded by the minimum and maximum distances of the rays, and the black shade in (b) shows the automatically obtained bounding volume.

RefiNeRF: Modelling dynamic neural radiance fields with inconsistent or missing camera parameters

“…Although refiNeRF shows promising results for both static and dynamic scenes, it shares the same limitations as the original NeRF approach, such as slow optimization and rendering, the requirement of dense 3D sampling, and dependence on heuristic coarse-to-fine scheduling strategies. Through the application of recent advancements such as iNGP [24] and vaxNeRF [17], we attempt to bypass some of the aforementioned limitations and believe that this framework will accelerate the widespread adoption of NeRFs.…”

A streak artifact reduction algorithm in sparse‐view CT using a self‐supervised neural representation

“…For example, Deng et al 54 implemented one of the neural representation 35 using a JAX library 55 and achieved 20-30 times faster training time compared to the official Tensorflow 56 implementation. Kondo et al 57 sped up the training by two to eight times by efficiently sampling the coordinated only the inside of the bounding volume (i.e., patient body in case of CT images). Tancik et al 58 reported that they could achieve the high-quality neural representation in 10-20 times less training iterations via meta-learned initialization 59,60 , which finds a good initial point of network parameters by optimizing them to several different scenes or objects alternately.…”

FPO++: efficient encoding and rendering of dynamic neural radiance fields by analyzing and enhancing Fourier PlenOctrees

“…Furthermore, PlenOctrees [72] use a hierarchical octree structure of the density and the viewdependent radiance in terms of spherical harmonics (SH) to entirely avoid network evaluations. Improving the convergence of the training process has also been investigated by using additional data such as depth maps [11] or a visual hull computed from binary foreground masks [24] as an additional guidance. Furthermore, meta learning approaches allow for a more effective initialization compared to random weights [55].…”