Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Abstract

3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction (from low-level geometry to high-level semantics including objects, places, rooms, buildings, etc.) and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory.This paper describes the first real-time Spatial Perception engINe (SPIN), a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance, to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed SPIN into a highly parallelized architecture, named Hydra, that combines fast early and midlevel perception processes (e.g., local mapping) with slower highlevel perception (e.g., global optimization of the scene graph). We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods, while running online.

Robot Localization Using Situational Graphs (S-Graphs) and Building Architectural Plans

“…The concept of 3D scene graphs for representing the structure of an environment in a hierarchical manner was introduced by [44] and [45]. [46] presented a hierarchical loop-closer detection in graphs. First, they compare descriptors in a top-down fashion to find putative loop closures at places, objects, and vision.…”

Situational Graphs for Robot Navigation in Structured Indoor Environments

“…Though promising in terms of scene representation, a major drawback of these models is that they do not tightly couple the estimate of the scene graph with the state estimate of the SLAM framework, to simultaneously optimize them, thus generating a scene graph and a SLAM graph in an independent manner. The very recent work Hydra [26] presents research in the direction of real-time scene graph generation as well as its optimization using loop closure constraints. Our approach is aligned in a similar direction towards real-time scene graphs.…”

Active Metric-Semantic Mapping by Multiple Aerial Robots

“…By contrast, other works build object-level maps using pre-collected 3D models [11], or basic shapes such as points [1], cuboids [4], ellipsoids [2], cylinders [7], a set of semantic keypoints [12], or a combination of 2D shapes [13]. Some efforts also account for ambiguity in data association [1], and the usage of semantic information in identifying moving objects [14], generating hierarchical descriptors for loop closure [15], and minimizing odometry drifts to assist long-range navigation [16]. However, these approaches do not define a measurement model that maps from state space to object classification confidence space, which is still an open and challenging problem [9].…”