Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understanding

Abstract

General scene understanding for robotics requires flexible semantic representation, so that novel objects and structures which may not have been known at training time can be identified, segmented and grouped. We present an algorithm which fuses general learned features from a standard pretrained network into a highly efficient 3D geometric neural field representation during real-time SLAM. The fused 3D feature maps inherit the coherence of the neural field's geometry representation. This means that tiny amounts of human labelling interacting at runtime enable objects or even parts of objects to be robustly and accurately segmented in an open set manner. Project page: https://makezur.github.io/ FeatureRealisticFusion/

Constructing Maps for Autonomous Robotics: An Introductory Conceptual Overview

“…The neural network is parametrized by spatial coordinates and pixel value estimates can be recovered by the integration along projection rays described in [41]. This system and its derivatives [42] use RGB-D inputs in tracking and mapping. Model architecture aside, this approach is otherwise analogous to more traditional ones like [21][22][23]35].…”

“…First, given semantic image segmentations (refer to Section 3.3 for more detail), the model can also be trained to output discrete [51] or open-set [52] semantic labels for each point in space. Second, taking advantage of the inherently projective nature of NRFs making recovery of depth information trivial, and the differentiable nature of neural networks, an NRF can be optimized along with the pose estimates and, thus, replace a traditional map in a real-time RGB-D SLAM system [37,42].…”

Neural Implicit Vision-Language Feature Fields

“…Neural implicit representations infer scene semantics [4], [5], [37]- [42] jointly with geometry using a multi-layer perceptron or similar parametric model. These have been extended to dynamic scenes [43].…”

“…These have been extended to dynamic scenes [43]. Neural feature fields [5], [38], [44], [45] are neural implicit representations which map continuous 3D coordinates to vector-valued features. Such representations have shown remarkable ability at scene segmentation and editing.…”

“…An environment representation that has wide applicability has several desirable properties, including: (1) can be built incrementally as the robot explores the environment, (2) enables real-time integration of new measurements, (3) has a compact memory footprint, (4) represents geometry at a high-level of detail, (5) is differentiable, (6) supports open set semantic queries, and (7) allows fast querying by downstream modules. Previously introduced 3D semantic scene representations are either built from global scene information [1], use closed set semantics [2]- [5], operate on a fixed level of detail [2], [3], or are not differentiable [2], [3]. In this paper, we take a step towards a representation which has the above-mentioned properties.…”