NICE-SLAM: Neural Implicit Scalable Encoding for SLAM

Zhu, Zihan; Peng, Songyou; Larsson, Viktor; Xu, Weiwei; Bao, Hujun; Cui, Zhaopeng; Oswald, Martin R.; Pollefeys, Marc

doi:10.1109/cvpr52688.2022.01245

Cited by 374 publications

(132 citation statements)

References 77 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…Sucar et al [9] propose iMAP, which demonstrates that an MLP can be used to represent the scene in simultaneous localization and mapping. Zhu et al [6] develop NICE-SLAM, which extends the idea of MLP-based scene representation to larger, multi-room environments. Ortiz et al [44] propose iSDF, which is a continual learning system for real-time signed distance field reconstruction.…”

Section: Related Workmentioning

confidence: 99%

“…Orthogonal to the camera pose estimation literature, advances in deep learning have led to a plethora of works investigating implicit shape and scene representations [5], [6], [7], [8], [9]. In particular, Neural Radiance Fields (NeRF) have gained significant popularity, as they can encode both 3D geometry and appearance of an environment [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

Maggio¹,

Abate²,

Shi³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present Loc-NeRF, a real-time vision-based robot localization approach that combines Monte Carlo localization and Neural Radiance Fields (NeRF). Our system uses a pre-trained NeRF model as the map of an environment and can localize itself in real-time using an RGB camera as the only exteroceptive sensor onboard the robot. While neural radiance fields have seen significant applications for visual rendering in computer vision and graphics, they have found limited use in robotics. Existing approaches for NeRF-based localization require both a good initial pose guess and significant computation, making them impractical for real-time robotics applications. By using Monte Carlo localization as a workhorse to estimate poses using a NeRF map model, Loc-NeRF is able to perform localization faster than the state of the art and without relying on an initial pose estimate. In addition to testing on synthetic data, we also run our system using real data collected by a Clearpath Jackal UGV and demonstrate for the first time the ability to perform real-time global localization with neural radiance fields. We make our code publicly available at https://github.com/MIT-SPARK/Loc-NeRF.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

Maggio¹,

Abate²,

Shi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…NeRFs provide an exciting new approach to scene representation in robotics, and a variety of use cases are currently explored. For example, NiceSLAM [5] and iMap [4] use a NeRF to represent a map within a SLAM system, while [6], [28] use it as a decoder within an autoencoder framework to learn an alternative representation for planning and reinforcement learning, and [7], [8] use NeRFs for data augmentation.…”

Section: A Neural Radiance Fieldsmentioning

confidence: 99%

“…As motivated in [20], we randomly split the available views from the individual datasets into 20% for training, and keep the remaining 80% for testing. This way, we can evaluate our model in a scenario where only a few (4)(5)(6)(7)(8)(9)(10)(11)(12) training views of a scene are available, which is more realistic for robotics applications than the dense viewpoint coverage typically encountered in NeRF datasets. Baselines: Using the established datasets for evaluation allows us to directly compare with S-NeRF [20] and the baseline results published in [20].…”

Section: A Experimental Setupmentioning

confidence: 99%

“…NeRFs and other implicit representations were met with an immense interest in the past two years, leading to an often-quoted "explosion" of work in this area [2]. While most of this work has been conducted by the computer graphics and vision communities, researchers in robotics have quickly started to explore possible use cases of NeRFs for important robotics tasks such as navigation [3], SLAM [4], [5], or manipulation [6]- [8]. Since the appearance of highly efficient NeRFs, such as Instant-NGP [9], that can be trained in seconds rather than many hours, the adoption of NeRFs for robotics has become palpable.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields

Sünderhauf¹,

Abou-Chakra²,

Miller³

2022

Preprint

View full text Add to dashboard Cite

We show that ensembling effectively quantifies model uncertainty in Neural Radiance Fields (NeRFs) if a density-aware epistemic uncertainty term is considered. The naive ensembles investigated in prior work simply average rendered RGB images to quantify the model uncertainty caused by conflicting explanations of the observed scene. In contrast, we additionally consider the termination probabilities along individual rays to identify epistemic model uncertainty due to a lack of knowledge about the parts of a scene unobserved during training. We achieve new state-of-the-art performance across established uncertainty quantification benchmarks for NeRFs, outperforming methods that require complex changes to the NeRF architecture and training regime. We furthermore demonstrate that NeRF uncertainty can be utilised for next-best view selection and model refinement.

show abstract

RLP‐VIO: Robust and lightweight plane‐based visual‐inertial odometry for augmented reality

Zhou

Yang

et al. 2022

Computer Animation & Virtual

View full text Add to dashboard Cite

We propose RLP-VIO-a robust and lightweight monocular visual-inertial odometry system using multiplane priors. With planes extracted from the point cloud, visual-inertial-plane PnP uses the plane information for fast localization. Depth estimation is susceptible to degenerated motion, so the planes are expanded in a reprojection consensus-based way robust to depth errors.For sensor fusion, our sliding-window optimization uses a novel structureless plane-distance error cost, which prevents the fill-in effect that poisons the BA problem's sparsity and permits the use of a smaller sliding window while maintaining good accuracy. The total computational cost is further reduced with our modified marginalization strategy. To further improve the tracking robustness, the landmark depths are constrained using the planes during degenerated motion. The whole system is parallelized with a three-stage pipeline. Under controlled environments, this parallelization runs deterministically and produces consistent results. The resulting VIO system is tested on widely used datasets and compared with several state-of-the-art systems. Our system achieves competitive accuracy and works robustly even on long and challenging sequences.To demonstrate the effectiveness of the proposed system, we also show the AR application running on mobile devices in real-time.

show abstract

NICE-SLAM: Neural Implicit Scalable Encoding for SLAM

Cited by 374 publications

References 77 publications

Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields

RLP‐VIO: Robust and lightweight plane‐based visual‐inertial odometry for augmented reality

Contact Info

Product

Resources

About