Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields

Abstract

Neural Radiance Fields (NeRF) is a popular view synthesis technique that represents a scene as a continuous volumetric function, parameterized by multilayer perceptrons that provide the volume density and view-dependent emitted radiance at each location. While NeRF-based techniques excel at representing fine geometric structures with smoothly varying view-dependent appearance, they often fail to accurately capture and reproduce the appearance of glossy surfaces. We address this limitation by introducing Ref-NeRF, which replaces NeRF's parameterization of view-dependent outgoing radiance with a representation of reflected radiance and structures this function using a collection of spatially-varying scene properties. We show that together with a regularizer on normal vectors, our model significantly improves the realism and accuracy of specular reflections. Furthermore, we show that our model's internal representation of outgoing radiance is interpretable and useful for scene editing.

Instant Neural Graphics Primitives with a Multiresolution Hash Encoding

“…However, it is frequently useful to input auxiliary dimensions 𝜉 ∈ R 𝐸 to the neural network, such as the view direction and material parameters when learning a light field. In such cases, the auxiliary dimensions can be encoded with established techniques whose cost does not scale superlinearly with dimensionality; we use the one-blob encoding [Müller et al 2019] in neural radiance caching ] and the spherical harmonics basis in NeRF, similar to concurrent work [Verbin et al 2021;Yu et al 2021a…”

PANDORA: Polarization-Aided Neural Decomposition Of Radiance

“…NeRF models the net outgoing radiance from a scene point in which both the material properties and the lighting are mixed. Several approaches such as NeRV [43], NeRD [7], NeuralPIL [8], PhySG [56], RefNeRF [46] have looked at decomposing this radiance into reflectance components and illumination. PhySG and NeuralPIL employ spherical Gaussian and data-driven embeddings to model the scene's illumination and reflectance.…”

“…For a certain object roughness, the obtained specular radiance involves integrating the specular BRDF along an incident direction factored by the incident illumination [6], which is a computationally expensive procedure that generally requires Monte Carlo. Inspired by [46], we instead use an IDE-based neural network to output the specular radiance, L D from the estimated roughness, α and surface normals, n. Moreover,on setting roughness close to zero, IllumNet also provides us the incident illumination, L i .…”

“…We see dark patches in the estimated illumination map where the network cannot correctly estimate the illumination due to self-occlusions. In future work, this limitation may be resolved using a similar method as Verbin et al [46], in which a learnable "bottleneck" vector is used to model the target features that are not explained by other parts of the network.…”

“…Implicit coordinate based networks are used to model the reflectance properties. Incident lighting is modelled as an implicit network with integrated directional embeddings [46]. We propose a differentiable rendering framework that takes as input the surface, reflectance parameters and illumination and renders polarization images under novel views.…”