RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

Abstract

Neural Radiance Fields (NeRF) have emerged as a powerful representation for the task of novel view synthesis due to their simplicity and state-of-the-art performance. Though NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available, its performance drops significantly when this number is reduced. We observe that the majority of artifacts in sparse input scenarios are caused by errors in the estimated scene geometry, and by divergent behavior at the start of training. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints, and annealing the ray sampling space during training. We additionally use a normalizing flow model to regularize the color of unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but in many cases also conditional models that are extensively pre-trained on large multi-view datasets.

SelfNeRF: Fast Training NeRF for Human from Monocular Self-rotating Video

“…Neural Radiance Field based Human Reconstruction NeRF(neural radiance field) [20] represents a static scene as a learnable 5D function and adopts volume rendering to render the image from any given view direction. Though vanilla NeRF only fits for static scenes, requires dense view inputs, and is slow to train and render, lots of work has been done to improve NeRF to dynamic scenes [26] and sparse view inputs [23] and increase the training and rendering speed [6]. Recently, some researchers have focused on applying the neural radiance field to human reconstruction.…”

“…Sample Space Annealing Following RegNeRF [23], we apply sample space annealing to avoid high-density values at ray origins. In practice, we start at a small range around the middle of the ray and gradually increase the sample range as training progresses.…”

E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context

“…Different from using existing functions, SPE [53] uses learnable spline functions and the latest instant-ngp [31] constructs a trainable hash map for shared embedding space. As for the regularization, many consistency constraints [16,8,32] regarding the 3D property have been studied in view synthesis. [46] regularize the in-domain initialization by a meta-learning approach.…”

Neural Mesh-Based Graphics

“…Some of the problems associated with NeRF [29] include higher computational cost and time of rendering complexity, the requirement of dense training views, the lack of across-scene generalization, and the need for test-time optimization. A number of works [3,13,18,26,30,41,60,63,64] have tried addressing these limitations. In particular, Spherical Harmonics [52] have been used to speed up inference of Neural Radiance Fields by factorizing the view-dependent appearance [63].…”