Omni-NeRF: Neural Radiance Field from 360° Image Captures

Gu, Kai; Maugey, Thomas; Knorr, Sebastian; Guillemot, Christine

doi:10.1109/icme52920.2022.9859817

Cited by 12 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multi-sphere [3] and multi-cylinder [15] images allocate blending weights and colors onto discrete surfaces, enabling the synthesis of free viewpoint panoramas. The Omni-NeRF [16] extends NeRF to be trained with raw fisheye captures. Some works [18,37] focus on photo-realistic renderings from a single ERP image with pre-acquired depth information.…”

Section: Omnidirectional Imagingmentioning

confidence: 99%

“…We first want to find the optimal settings for combining the hashencoding (HE) and frequency-encoding (FE) levels. To achieve this, we performed a parameter sweep across different frequency levels (2,4,8,16) while keeping the levels of the hash grid to be fixed at the default value of 16. We conduct our experiment on the synthetic datasets of ODIs from the 2 Blender demos ("Classroom", "Lone Monk") [1].…”

Section: Hash-frequency Encodingmentioning

confidence: 99%

See 1 more Smart Citation

Vanishing Point Aided Hash-Frequency Encoding for Neural Radiance Fields (NeRF) from Sparse 360°Input

Gu,

Maugey,

Knorr

et al. 2023

2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

Self Cite

View full text Add to dashboard Cite

Section: Omnidirectional Imagingmentioning

confidence: 99%

Section: Hash-frequency Encodingmentioning

confidence: 99%

Vanishing Point Aided Hash-Frequency Encoding for Neural Radiance Fields (NeRF) from Sparse 360°Input

Gu,

Maugey,

Knorr

et al. 2023

2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

Self Cite

View full text Add to dashboard Cite

“…OmniNeRF [44] synthesizes novel fish-eye projection images, using spherical sampling to improve the quality of results. 360Roam [45] is a scene-level NeRF system that can synthesize images of large-scale indoor scenes in real-time and support VR roaming.…”

Section: B 360 Panorama View Synthesismentioning

confidence: 99%

360FusionNeRF: Panoramic Neural Radiance Fields with Joint Guidance

Kulkarni¹,

Yin²,

Scherer³

2022

Preprint

View full text Add to dashboard Cite

We present a method to synthesize novel views from a single 360 • panorama image based on the neural radiance field (NeRF). Prior studies in a similar setting rely on the neighborhood interpolation capability of multi-layer perceptions to complete missing regions caused by occlusion, which leads to artifacts in their predictions. We propose 360FusionNeRF, a semi-supervised learning framework where we introduce geometric supervision and semantic consistency to guide the progressive training process. Firstly, the input image is re-projected to 360 • images and auxiliary depth maps are extracted at other camera positions. The depth supervision, in addition to the NeRF color guidance, improves the geometry of the synthesized views. Additionally, we introduce a semantic consistency loss that encourages realistic renderings of novel views. We extract these semantic features using a pre-trained visual encoder such as CLIP, a Vision Transformer trained on hundreds of millions of diverse 2D photographs mined from the web with natural language supervision. Experiments indicate that our proposed method can produce plausible completions of unobserved regions while preserving the features of the scene. When trained across various scenes, 360FusionNeRF consistently achieves the state-of-the-art performance when transferring to synthetic Structured3D dataset (PSNR ∼ 5%, SSIM ∼3% LPIPS ∼13%), real-world Matterport3D dataset (PSNR ∼3%, SSIM ∼3% LPIPS ∼9%) and Replica360 dataset (PSNR ∼8%, SSIM ∼2% LPIPS ∼18%). We provide the source code at https://github.com/MetaSLAM/360FusionNeRF.

show abstract

“…This function maps a 5D input -3D spatial coordinates and 2D viewing directions -into a viewdependent RGB triplet and one value corresponding to the volume density. NeRF's success has led to several extension works targeting its limitations, such as generalization [2], relighting [3], and different imaging input types [4], [5].…”

Section: Introductionmentioning

confidence: 99%

Compression of PlenOctree Model Attributes Enabling Fast Communication and Rendering of Neural Radiance Fields

Freitas,

Guillemot,

Tabus

2023

2023 31st European Signal Processing Conference (EUSIPCO)

Self Cite

View full text Add to dashboard Cite

Neural Radiance Fields (NeRF) have led the advancement of techniques for 3D scene representations for synthesizing views through the use of a multilayer perceptron. Its inability to achieve real-time performance for photorealistic image rendering, however, has enabled the advent of methods that speed-up rendering time at the cost of rendering quality or model complexity. In this work, we focus on the existing Plenoctree method, which possesses high rendering speed but unfortunately needs a large space for storing and transmitting its model. We address this weakness by proposing improvements to the different blocks of its pipeline and adding an efficient compression stage, without modifying the underlying representation, while maintaining high rendering quality and speed. Results over a set of test camera poses -using our methods which were obtained with data from the training and validation datasets -show that we can reduce about eight times the bit rates of the encoded models and still obtain a higher quality of the synthesized images when comparing them to the original PlenOctree models or, alternatively, a reduction of about 50 times while presenting minimal degradation for novel view synthesis.

show abstract

Omni-NeRF: Neural Radiance Field from 360° Image Captures

Cited by 12 publications

References 17 publications

Vanishing Point Aided Hash-Frequency Encoding for Neural Radiance Fields (NeRF) from Sparse 360°Input

Vanishing Point Aided Hash-Frequency Encoding for Neural Radiance Fields (NeRF) from Sparse 360°Input

360FusionNeRF: Panoramic Neural Radiance Fields with Joint Guidance

Compression of PlenOctree Model Attributes Enabling Fast Communication and Rendering of Neural Radiance Fields

Contact Info

Product

Resources

About