Neural Volume Rendering: NeRF And Beyond

Dellaert, Frank; Yen-Chen, Lin

doi:10.48550/arxiv.2101.05204

Cited by 19 publications

(21 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…uses integrated positional encoding over a cone frustum to render anti-aliased multi-scale images. NeRF has opened the door to many exciting research directions [73,14,1,90,60,72,56,84,39,59,17,19,21,44,67,4]. However, NeRF (and closely-related variants) are slow to train and render, as rendering a single pixel requires evaluating hundreds of samples along a ray, each constituting an expensive neural network query.…”

Section: Related Workmentioning

confidence: 99%

RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis

Tremblay¹,

Meshry²,

Evans³

et al. 2022

Preprint

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis

Tremblay¹,

Meshry²,

Evans³

et al. 2022

Preprint

View full text Add to dashboard Cite

“…Recent studies have been focusing on model implicit functions with multi-layer perceptron (MLP) due to its expressive representation power and low memory foot-print, including scene (geometry & appearance) representation [10,23,22,26,28,33,40,31,39] and free-view rendering [37,18,34,29,19,24,17,44,38,3]. In particular, NeRF [24] has opened up a line of research (see [7] for an overview) combining neural implicit functions together with volume rendering to achieve photo-realistic rendering results. However, it is non-trivial to find a proper threshold to extract surfaces from the predicted density, and the recovered geometry is far from satisfactory.…”

Section: Related Workmentioning

confidence: 99%

Volume Rendering of Neural Implicit Surfaces

Yariv¹,

Gu²,

Kasten³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural volume rendering became increasingly popular recently due to its success in synthesizing novel views of a scene from a sparse set of input images. So far, the geometry learned by neural volume rendering techniques was modeled using a generic density function. Furthermore, the geometry itself was extracted using an arbitrary level set of the density function leading to a noisy, often low fidelity reconstruction. The goal of this paper is to improve geometry representation and reconstruction in neural volume rendering. We achieve that by modeling the volume density as a function of the geometry. This is in contrast to previous work modeling the geometry as a function of the volume density. In more detail, we define the volume density function as Laplace's cumulative distribution function (CDF) applied to a signed distance function (SDF) representation. This simple density representation has three benefits: (i) it provides a useful inductive bias to the geometry learned in the neural volume rendering process; (ii) it facilitates a bound on the opacity approximation error, leading to an accurate sampling of the viewing ray. Accurate sampling is important to provide a precise coupling of geometry and radiance; and (iii) it allows efficient unsupervised disentanglement of shape and appearance in volume rendering. Applying this new density representation to challenging scene multiview datasets produced high quality geometry reconstructions, outperforming relevant baselines. Furthermore, switching shape and appearance between scenes is possible due to the disentanglement of the two.Preprint. Under review.

show abstract

“…We go over the NeRF basics and introduce visual hull in this section. For more in-depth summaries of NeRFs and extensions, please refer to [11].…”

Section: Preliminariesmentioning

confidence: 99%

VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field

Kondo¹,

Ikeda²,

Tagliasacchi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural Radiance Field (NeRF) is a popular method in data-driven 3D reconstruction. Given its simplicity and high quality rendering, many NeRF applications are being developed. However, NeRF's big limitation is its slow speed. Many attempts are made to speeding up NeRF training and inference, including intricate code-level optimization and caching, use of sophisticated data structures, and amortization through multi-task and meta learning. In this work, we revisit the basic building blocks of NeRF through the lens of classic techniques before NeRF. We propose Voxel-Accelearated NeRF (VaxNeRF), integrating NeRF with visual hull, a classic 3D reconstruction technique only requiring binary foreground-background pixel labels per image. Visual hull, which can be optimized in about 10 seconds, can provide coarse in-out field separation to omit substantial amounts of network evaluations in NeRF. We provide a clean fully-pythonic, JAX-based implementation on the popular JaxNeRF codebase, consisting of only about 30 lines of code changes and a modular visual hull subroutine, and achieve about 2-8x faster learning on top of the highly-performative JaxNeRF baseline with zero degradation in rendering quality. With sufficient compute, this effectively brings down full NeRF training from hours to 30 minutes. We hope VaxNeRF -a careful combination of a classic technique with a deep method (that arguably replaced it) -can empower and accelerate new NeRF extensions and applications, with its simplicity, portability, and reliable performance gains. Codes are available at https://github.com/naruya/VaxNeRF. (a) NeRF (b) VaxNeRF Figure 1. NeRF (a) uses hierarchical sampling, i.e., a mixture of coarse (pink) and fine (light blue) sampling, to estimate the color from a specified viewpoint. Instead, VaxNeRF (b) samples only the inside of the bounding volume uniformly and densely. The orange circle shows the area bounded by the minimum and maximum distances of the rays, and the black shade in (b) shows the automatically obtained bounding volume.

show abstract

Neural Volume Rendering: NeRF And Beyond

Cited by 19 publications

References 29 publications

RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis

RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis

Volume Rendering of Neural Implicit Surfaces

VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field

Contact Info

Product

Resources

About