NeX: Real-time View Synthesis with Neural Basis Expansion

Wizadwongsa, Suttisak; Phongthawee, Pakkapon; Yenphraphai, Jiraphon; Suwajanakorn, Supasorn

doi:10.1109/cvpr46437.2021.00843

Cited by 232 publications

(142 citation statements)

References 37 publications

Supporting

Mentioning

142

Contrasting

Order By: Relevance

“…surface and voxel) of 3D shapes that reconstructs neural radiance fields from singleview RGB image and have not investigated that from more challenging partial geometries (e.g. depth maps or multiplane images [55,60]). In future work, we plan to extend our method to such partial geometries, thereby making our method more flexible.…”

Section: Discussionmentioning

confidence: 99%

PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

Yu¹,

Tang²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present PVSeRF, a learning framework that reconstructs neural radiance fields from single-view RGB images, for novel view synthesis. Previous solutions, such as pix-elNeRF [66], rely only on pixel-aligned features and suffer from feature ambiguity issues. As a result, they struggle with the disentanglement of geometry and appearance, leading to implausible geometries and blurry results. To address this challenge, we propose to incorporate explicit geometry reasoning and combine it with pixel-aligned features for radiance field prediction. Specifically, in addition to pixel-aligned features, we further constrain the radiance field learning to be conditioned on i) voxel-aligned features learned from a coarse volumetric grid and ii) fine surfacealigned features extracted from a regressed point cloud. We show that the introduction of such geometry-aware features helps to achieve a better disentanglement between appearance and geometry, i.e. recovering more accurate geometries and synthesizing higher quality images of novel views. Extensive experiments against state-of-the-art methods on ShapeNet benchmarks demonstrate the superiority of our approach for single-image novel view synthesis.

show abstract

Section: Discussionmentioning

confidence: 99%

PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

Yu¹,

Tang²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…8), applying our method to unbounded and forward-facing scenes optimally requires further work as the data distribution is different for unbounded scenes. The forward-facing scenes inherently do not support 6-DOF viewing, and we suggest MPIs may be more appropriate in this case [56].…”

Section: Discussionmentioning

confidence: 93%

“…Octrees have been used in recent work to decrease the memory requirements during training for other 3D tasks [36,10,49,54]. Concurrent with this work, Nex [56] extends MPIs to encode spherical basis functions that enable view-dependent rendering effects in real-time. However, unlike our representation, their approach is limited in the viewing direction due to their use of MPIs.…”

Section: Related Workmentioning

confidence: 99%

PlenOctrees for Real-time Rendering of Neural Radiance Fields

Yu¹,

Li²,

Tancik³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800×800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve viewdependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu. net/plenoctrees.

show abstract

“…The resulting mesh geometry can be effectively rendered using standard graphics engines, making the approach attractive for mobile 3D photography. At the same time, while being efficient, our scene representation has only limited ability to represent view-dependent effects, and an extension similar to NeX [39] would be required to improve this ability.…”

Section: Summary and Discussionmentioning

confidence: 99%

Stereo Magnification with Multi-Layer Images

Khakhulin¹,

Denis²,

Solovev³

et al. 2022

Preprint

View full text Add to dashboard Cite

Representing scenes with multiple semi-transparent colored layers has been a popular and successful choice for real-time novel view synthesis. Existing approaches infer colors and transparency values over regularly-spaced layers of planar or spherical shape. In this work, we introduce a new view synthesis approach based on multiple semi-transparent layers with scene-adapted geometry. Our approach infers such representations from stereo pairs in two stages. The first stage infers the geometry of a small number of data-adaptive layers from a given pair of views. The second stage infers the color and the transparency values for these layers producing the final representation for novel view synthesis. Importantly, both stages are connected through a differentiable renderer, and are trained in an end-to-end manner. In the experiments, we demonstrate the advantage of the proposed approach over the use of regularly-spaced layers with no adaptation to scene geometry. Despite being orders of magnitude faster during rendering, our approach also outperforms a recently proposed IBRNet system based on implicit geometry representation. See results at samsunglabs.github.io/StereoLayers.

show abstract

NeX: Real-time View Synthesis with Neural Basis Expansion

Cited by 232 publications

References 37 publications

PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

PlenOctrees for Real-time Rendering of Neural Radiance Fields

Stereo Magnification with Multi-Layer Images

Contact Info

Product

Resources

About