Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs

Abstract

Humans are able to form a complex mental model of the environment they move in. This mental model captures geometric and semantic aspects of the scene, describes the environment at multiple levels of abstractions (e.g., objects, rooms, buildings), includes static and dynamic entities and their relations (e.g., a person is in a room at a given time). In contrast, current robots' internal representations still provide a partial and fragmented understanding of the environment, either in the form of a sparse or dense set of geometric primitives (e.g., points, lines, planes, voxels), or as a collection of objects. This paper attempts to reduce the gap between robot and human perception by introducing a novel representation, a 3D Dynamic Scene Graph (DSG), that seamlessly captures metric and semantic aspects of a dynamic environment. A DSG is a layered graph where nodes represent spatial concepts at different levels of abstraction, and edges represent spatio-temporal relations among nodes. Our second contribution is Kimera, the first fully automatic method to build a DSG from visual-inertial data. Kimera includes state-of-the-art techniques for visual-inertial SLAM, metric-semantic 3D reconstruction, object localization, human pose and shape estimation, and scene parsing. Our third contribution is a comprehensive evaluation of Kimera in real-life datasets and photo-realistic simulations, including a newly released dataset, uHumans2, which simulates a collection of crowded indoor and outdoor scenes. Our evaluation shows that Kimera achieves state-of-the-art performance in visual-inertial SLAM, estimates an accurate 3D metric-semantic mesh model in real-time, and builds a DSG of a complex indoor environment with tens of objects and humans in minutes. Our final contribution is to showcase how to use a DSG for real-time hierarchical semantic path-planning. The core modules in Kimera have been released open source.

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

“…Armeni et al [4] model the environment as a graph including low-level geometry (i.e., a metric-semantic mesh), objects, rooms, and camera locations. Rosinol et al [49,50] augment the model with a topological map of places, as well as a layer describing moving entities in the environment. The approaches in [4,49,50] are designed for offline operation.…”

“…Rosinol et al [49,50] augment the model with a topological map of places, as well as a layer describing moving entities in the environment. The approaches in [4,49,50] are designed for offline operation. Other papers focus on reconstructing a graph of objects and their relations [26,63,67].…”

“…After extracting the 3D mesh within the active window, we segment objects by performing Euclidean clustering of the 3D metric-semantic mesh vertices; in particular, we independently cluster vertices belonging to each semantic class. As in [50], the result of the Euclidean clustering is then used to estimate a centroid and bounding box for each putative object. These are then associated with previous object nodes inside the active window, and any putative objects that do not correspond to a previous object node are added as new nodes.…”

“…Datasets. We utilize two datasets for our experiments: uHu-mans2 (uH2) [50] and SidPac. The uH2 dataset is a Unitybased simulated dataset [50] that includes three scenes: a small apartment, an office, and a subway station.…”

“…Unfortunately, ESDFs scale poorly in the size of the environment [43]. Moreover, the extraction of places and rooms in [49,50] involves batch algorithms that process the entire ESDF, whose computational cost grows over time and is incompatible with real-time operation. Finally, the ESDF is reconstructed from the robot trajectory estimate; however, during operation the trajectory estimate keeps changing in response to loop closures; therefore, the approaches in [49,50] would need to rebuild the scene graph from scratch after every loop closure, which again clashes with real-time operation.…”