FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality

Deng, Nianchen; He, Zhenyi; Ye, Jiannan; Duinkharjav, Budmonde; Chakravarthula, Praneeth; Yang, Xubo; Sun, Qi

doi:10.1109/tvcg.2022.3203102

Cited by 62 publications

(15 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Most recent developments of NeRF include the navigation of neural renderings in Virtual Reality (VR) or Extended Reality (XR) applications (Deng et al, 2022;Park et al, 2022).…”

Section: Nerf Rendering For Virtual and Extended Realitymentioning

confidence: 99%

Neural Radiance Fields (Nerf): Review and Potential Applications to Digital Cultural Heritage

Croce

Crepaldi

Luca

et al. 2023

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

Abstract. Neural Radiance Fields (NeRF or NeRFs) are to date emerging as a novel method for synthesizing novel views of complex 3D scenes, leveraging an artificial neural network to optimize a volumetric scene function using a set of input views. We conduct a preliminary critical review of the scientific and technical literature on NeRFs, and we highlight possible applications of the latter in the Cultural Heritage domain, for the image-based reconstruction of 3D models of real, multi-scale objects, even in combination with the more well-established photogrammetric techniques. A comparison is made between NeRFs and photogrammetry in terms of operating procedures and outputs (volumetric renderings vs. point clouds or meshes). It is demonstrated that NeRFs could be conveniently used for rendering objects (sculptures, archaeological remains, sites, paintings etc.) that are challenging for photogrammetry, typically: i) metallic, translucent, and/or transparent surfaces; ii) objects that present homogeneous textures; iii) occlusions, vegetation, and elements of very fine detail.

show abstract

“…Most recent developments of NeRF include the navigation of neural renderings in Virtual Reality (VR) or Extended Reality (XR) applications (Deng et al, 2022;Park et al, 2022).…”

Section: Nerf Rendering For Virtual and Extended Realitymentioning

confidence: 99%

Neural Radiance Fields (Nerf): Review and Potential Applications to Digital Cultural Heritage

Croce

Crepaldi

Luca

et al. 2023

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…The continuous nature of INRs is particularly appealing when dealing with irregularly sampled signals such as a point clouds. Since its first widespread usage in novel view synthesis in graphics [29], INRs have pervaded nearly all fields of vision and signal processing including rendering [24], computational imaging [6,12], medical imaging [51], and virtual reality [16].…”

Section: Implicit Representations Inrs Are Continuous Learned Functio...mentioning

confidence: 99%

WIRE: Wavelet Implicit Neural Representations

Saragadam¹,

LeJeune²,

Tan³

et al. 2023

Preprint

View full text Add to dashboard Cite

Implicit neural representations (INRs) have recently advanced numerous vision-related areas. INR performance depends strongly on the choice of the nonlinear activation function employed in its multilayer perceptron (MLP) network. A wide range of nonlinearities have been explored, but, unfortunately, current INRs designed to have high accuracy also suffer from poor robustness (to signal noise, parameter variation, etc.). Inspired by harmonic analysis, we develop a new, highly accurate and robust INR that does not exhibit this tradeoff. Wavelet Implicit neural REpresentation (WIRE) uses a continuous complex Gabor wavelet activation function that is well-known to be optimally concentrated in space-frequency and to have excellent biases for representing images. A wide range of experiments (image denoising, image inpainting, super-resolution, computed tomography reconstruction, image overfitting, and novel view synthesis with neural radiance fields) demonstrate that WIRE defines the new state of the art in INR accuracy, training time, and robustness.

show abstract

“…Foveated graphics techniques exploit eccentricity-dependent aspects of human vision, such as acuity, to minimize the bandwidth of a graphics system by optimizing bit depth [McCarthy et al 2004], color-fidelity , level-of-detail [Luebke and Hallen 2001;Murphy and Duchowski 2001], or by simply reducing the number of vertices or fragments a graphics processing unit has to sample, ray-trace, shade, or transmit to the display; see [Duchowski et al 2004;Koulieris et al 2019] for a review of this area. Foveated rendering is perhaps the most well-known example of this class of algorithms [Deng et al 2022;Friston et al 2019;Geisler and Perry 1998;Guenter et al 2012;Kaplanyan et al 2019;Patney et al 2016;Sun et al 2017;Tariq et al 2022;Tursun et al 2019]. The perceptual models underlying foveated graphics usually exploit spatial aspects of eccentricity-dependent human vision but, to the best of our knowledge, none of them model cognitive or attentional effects of human vision, which we aim to address with our work.…”

Section: Related Work 21 Foveated Graphicsmentioning

confidence: 99%