Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

Chibane, Julian; Bansal, Aayush; Lazova, Verica; Pons-Moll, Gerard

doi:10.1109/cvpr46437.2021.00782

Cited by 169 publications

(68 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite synthesizing highfidelity novel views, these methods require longstanding optimization process and cannot generalize to new scenes. 2) the second track attempts to learn generalize neural radiance field across multiple scenes [3,6,54,57,66]. Among this, pixelNeRF [66] is the most relevant method to ours, which Given a single input image, we first 1) extract the spatial feature map using a fully convolutional image encoder E, 2) learn a volumetric grid through volume generator G V , and 3) regress a surface point set of the object through a point set generator GS .…”

Section: Novel View Synthesis and Neural Radiance Fieldmentioning

confidence: 99%

“…learns the scene priors conditioned on the pixel-aligned features, and can switch to new scenes flexibly. Although other methods [3,6,54,57] can also be applied to novel scene through a single forward pass, their methods are equipped to multiple input views, while we focus on the more challenging single-view input setting.…”

Section: Novel View Synthesis and Neural Radiance Fieldmentioning

confidence: 99%

See 1 more Smart Citation

PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

Yu¹,

Tang²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present PVSeRF, a learning framework that reconstructs neural radiance fields from single-view RGB images, for novel view synthesis. Previous solutions, such as pix-elNeRF [66], rely only on pixel-aligned features and suffer from feature ambiguity issues. As a result, they struggle with the disentanglement of geometry and appearance, leading to implausible geometries and blurry results. To address this challenge, we propose to incorporate explicit geometry reasoning and combine it with pixel-aligned features for radiance field prediction. Specifically, in addition to pixel-aligned features, we further constrain the radiance field learning to be conditioned on i) voxel-aligned features learned from a coarse volumetric grid and ii) fine surfacealigned features extracted from a regressed point cloud. We show that the introduction of such geometry-aware features helps to achieve a better disentanglement between appearance and geometry, i.e. recovering more accurate geometries and synthesizing higher quality images of novel views. Extensive experiments against state-of-the-art methods on ShapeNet benchmarks demonstrate the superiority of our approach for single-image novel view synthesis.

show abstract

Section: Novel View Synthesis and Neural Radiance Fieldmentioning

confidence: 99%

Section: Novel View Synthesis and Neural Radiance Fieldmentioning

confidence: 99%

PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis

Yu¹,

Tang²,

Qin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The idea of projecting information from source views to 3D then using the neural radiance field framework to render a target view has also been used in learning-based approaches [3,4,6,33,35,43]. These approaches train on multiple scenes networks that take as input features from the source views aggregated at a given 3D point and output radiance and occupancy for this point.…”

Section: Related Workmentioning

confidence: 99%

Improving neural implicit surfaces geometry with patch warping

Darmon¹,

Bascle²,

Devaux³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural implicit surfaces have become an important technique for multi-view 3D reconstruction but their accuracy remains limited. In this paper, we argue that this comes from the difficulty to learn and render high frequency textures with neural networks. We thus propose to add to the standard neural rendering optimization a direct photo-consistency term across the different views. Intuitively, we optimize the implicit geometry so that it warps views on each other in a consistent way. We demonstrate that two elements are key to the success of such an approach: (i) warping entire patches, using the predicted occupancy and normals of the 3D points along each ray, and measuring their similarity with a robust structural similarity (SSIM); (ii) handling visibility and occlusion in such a way that incorrect warps are not given too much importance while encouraging a reconstruction as complete as possible. We evaluate our approach, dubbed NeuralWarp, on the standard DTU and EPFL benchmarks and show it outperforms state of the art unsupervised implicit surfaces reconstructions by over 20% on both datasets. Our code is available at https://github.com/fdarmon/NeuralWarp

show abstract

“…Bemana et al [5] works in static settings but predicts not only the radiance field but also lighting given varying illumination data. Chibane et al [14] trade instant depth predictions and synthesis for the requirement of multiple images. Alternatively, volumetric representations [39,40] can also being utilized for capturing dynamic scenes.…”

Section: Related Workmentioning

confidence: 99%