GeoNeRF: Generalizing NeRF with Geometry Priors

Johari, Mohammad Mahdi; Lepoittevin, Yann; Fleuret, François

doi:10.48550/arxiv.2111.13539

Cited by 1 publication

(2 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Contrastingly, MVSNeRF [9] trains on DTU [23] and tests on held out DTU scenes, real forward-facing dataset (RFF) [37], and Blender [37]. Various other works [24,31,66] have explored different experimental setups. In this work, in an attempt to fairly evaluate against prior works, we use two experimental settings.…”

Section: Resultsmentioning

confidence: 99%

“…In contrast with these works, our method 1) does not require deep convolutional features, operating directly on linear projections of local patches, similarly to ViT [14]; 2) does not require volume rendering, producing the final colors directly from a reference set of patches; and 3) is independent of the input frame of reference, leveraging canonicalized ray, point and camera representations, which improves its generalization ability. Concurrent work on neural rendering generalizable to unseen scenes include GeoNeRF [24] and NeuRay [31], but both require at least partial depth maps during training.…”

Section: Image-based Renderingmentioning

confidence: 99%

See 1 more Smart Citation

Generalizable Patch-Based Neural Rendering

Suhail¹,

Esteves²,

Sigal³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural rendering has received tremendous attention since the advent of Neural Radiance Fields (NeRF), and has pushed the state-ofthe-art on novel-view synthesis considerably. The recent focus has been on models that overfit to a single scene, and the few attempts to learn models that can synthesize novel views of unseen scenes mostly consist of combining deep convolutional features with a NeRF-like model. We propose a different paradigm, where no deep visual features and no NeRF-like volume rendering are needed. Our method is capable of predicting the color of a target ray in a novel scene directly, just from a collection of patches sampled from the scene. We first leverage epipolar geometry to extract patches along the epipolar lines of each reference view. Each patch is linearly projected into a 1D feature vector and a sequence of transformers process the collection. For positional encoding, we parameterize rays as in a light field representation, with the crucial difference that the coordinates are canonicalized with respect to the target ray, which makes our method independent of the reference frame and improves generalization. We show that our approach outperforms the state-of-the-art on novel view synthesis of unseen scenes even when being trained with considerably less data than prior work. Our code is available at https://mohammedsuhail.net/gen_patch_neural_rendering/.

show abstract