SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences

Wu, Shuncheng; Wald, Johanna; Tateno, Keisuke; Navab, Nassir; Tombari, Federico

doi:10.1109/cvpr46437.2021.00743

Cited by 92 publications

(68 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approaches in [4,49,50] are designed for offline operation. Other papers focus on reconstructing a graph of objects and their relations [26,63,67]. Wu et al [67] predict objects and relations in real-time using a graph-neural network.…”

Section: Related Workmentioning

confidence: 99%

“…3D Scene Graphs [4,26,49,50,63,67] have recently emerged as powerful high-level representations of 3D environments. A 3D scene graph (Fig.…”

Section: Introductionmentioning

confidence: 99%

“…While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time remains uncharted territory. The works [26,63,67] allow real-time operation but are restricted to "flat" 3D scene graphs, and are mostly concerned with objects and their relations, while disregarding the top layers in Fig. 1.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hughes¹,

Chang²,

Carlone³

2022

Preprint

View full text Add to dashboard Cite

3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction (from low-level geometry to high-level semantics including objects, places, rooms, buildings, etc.) and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory.This paper describes the first real-time Spatial Perception engINe (SPIN), a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance, to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed SPIN into a highly parallelized architecture, named Hydra, that combines fast early and midlevel perception processes (e.g., local mapping) with slower highlevel perception (e.g., global optimization of the scene graph). We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods, while running online.

show abstract

Section: Related Workmentioning

confidence: 99%

“…3D Scene Graphs [4,26,49,50,63,67] have recently emerged as powerful high-level representations of 3D environments. A 3D scene graph (Fig.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hughes¹,

Chang²,

Carlone³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, their focus is on efficient annotation and collection of such graphs from 3D sensors. Similarly, more recent efforts such as (Zhang et al 2021;Wu et al 2021) are also targeted at improvising the efficiency of constructing a 3D scene graph from RGBD scans, while our focus is on constructing pseudo-3D graphs leveraging recent advancements in 2D-to-3D methods. We note that while precise 3D scene graphs may be important for several tasks such as robot navigation or manipulation, they need not be required for reasoning tasks such as what we consider in this paper, and for such tasks approximate 3D reasoning may be sufficient.…”

Section: Related Workmentioning

confidence: 99%

(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering

Cherian¹,

Hori²,

Marks³

et al. 2022

Preprint

View full text Add to dashboard Cite

Spatio-temporal scene-graph approaches to video-based reasoning tasks such as video question-answering (QA) typically construct such graphs for every video frame. Such approaches often ignore the fact that videos are essentially sequences of 2D "views" of events happening in a 3D space, and that the semantics of the 3D scene can thus be carried over from frame to frame. Leveraging this insight, we propose a (2.5+1)D scene graph representation to better capture the spatio-temporal information flows inside the videos. Specifically, we first create a 2.5D (pseudo-3D) scene graph by transforming every 2D frame to have an inferred 3D structure using an off-the-shelf 2D-to-3D transformation module, following which we register the video frames into a shared (2.5+1)D spatio-temporal space and ground each 2D scene graph within it. Such a (2.5+1)D graph is then segregated into a static sub-graph and a dynamic sub-graph, corresponding to whether the objects within them usually move in the world. The nodes in the dynamic graph are enriched with motion features capturing their interactions with other graph nodes. Next, for the video QA task, we present a novel transformerbased reasoning pipeline that embeds the (2.5+1)D graph into a spatio-temporal hierarchical latent space, where the sub-graphs and their interactions are captured at varied granularity. To demonstrate the effectiveness of our approach, we present experiments on the NExT-QA and AVSD-QA datasets. Our results show that our proposed (2.5+1)D representation leads to faster training and inference, while our hierarchical model showcases superior performance on the video QA task versus the state of the art.

show abstract

“…An important step in semantic segmentation is feature extraction. Based on the methods used for feature extraction, semantic segmentation approaches can be roughly categorized into three groups: handcrafted-feature-based [8,14,39,46,48,[50][51][52], learning-based [16,24,37,38,47], and hybrid methods [23,53]. Handcrafted features are often effective when with limited training data.…”

Section: Semantic Segmentation Of 3d Datamentioning

confidence: 99%

PSSNet: Planarity-sensible Semantic Segmentation of Large-scale Urban Meshes

Gao¹,

Nan²,

Boom³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce a novel deep learning-based framework to interpret 3D urban scenes represented as textured meshes. Based on the observation that object boundaries typically align with the boundaries of planar regions, our framework achieves semantic segmentation in two steps: planaritysensible over-segmentation followed by semantic classification. The over-segmentation step generates an initial set of mesh segments that capture the planar and non-planar regions of urban scenes. In the subsequent classification step, we construct a graph that encodes geometric and photometric features of the segments in its nodes and multiscale contextual features in its edges. The final semantic segmentation is obtained by classifying the segments using a graph convolutional network. Experiments and comparisons on a large semantic urban mesh benchmark demonstrate that our approach outperforms the state-ofthe-art methods in terms of boundary quality and mean IoU (intersection over union). Besides, we also introduce several new metrics for evaluating mesh over-segmentation methods dedicated for semantic segmentation, and our proposed over-segmentation approach outperforms stateof-the-art methods on all metrics. Our source code will be released when the paper is accepted.

show abstract

SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences

Cited by 92 publications

References 51 publications

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering

PSSNet: Planarity-sensible Semantic Segmentation of Large-scale Urban Meshes

Contact Info

Product

Resources

About