Digging into Radiance Grid for Real-Time View Synthesis with Detail Preservation

Abstract

Neural Radiance Fields (NeRF) [30] series are impressive in representing scenes and synthesizing high-quality novel views. However, most previous works fail to preserve texture details and suffer from slow training speed. A recent method SNeRG [10] demonstrates that baking a trained NeRF as a Sparse Neural Radiance Grid enables real-time view synthesis with slight scarification of rendering quality. In this paper, we dig into the Radiance Grid representation and present a set of improvements, which together result in significantly boosted performance in terms of both speed and quality. First, we propose an HieRarchical Sparse Radiance Grid (HrSRG) representation that has higher voxel resolution for informative spaces and fewer voxels for other spaces. HrSRG leverages a hierarchical voxel grid building process inspired by [29], and can describe a scene at high resolution without excessive memory footprint. Furthermore, we show that directly optimizing the voxel grid leads to surprisingly good texture details in rendered images. This direct optimization is memory-friendly and requires multiple orders of magnitude less time than conventional NeRFs as it only involves a tiny MLP. Finally, we find that a critical factor that prevents fine details restoration is the misaligned 2D pixels among images caused by camera pose errors. We propose to use the perceptual loss to add tolerance to misalignments, leading to the improved visual quality of rendered images.

MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes

“…Other methods addressed real-time rendering by precomputing and storing (i.e. baking) NeRF's view-dependent colors and opacities in volumetric data structures [Garbin et al 2021;Hedman et al 2021;Yu et al 2021;Zhang et al 2022], or by splitting the scene into voxels and representing each voxel with a small separate MLP [Reiser et al 2021]. However, these representations consume a lot of graphics memory and are thus limited to objects, not scenes.…”