SemanticFusion: Dense 3D semantic mapping with convolutional neural networks

Abstract

Ever more robust, accurate and detailed mapping using visual sensing has proven to be an enabling factor for mobile robots across a wide variety of applications. For the next level of robot intelligence and intuitive user interaction, maps need extend beyond geometry and appearence -they need to contain semantics. We address this challenge by combining Convolutional Neural Networks (CNNs) and a state of the art dense Simultaneous Localisation and Mapping (SLAM) system, ElasticFusion, which provides long-term dense correspondence between frames of indoor RGB-D video even during loopy scanning trajectories. These correspondences allow the CNN's semantic predictions from multiple view points to be probabilistically fused into a map. This not only produces a useful semantic 3D map, but we also show on the NYUv2 dataset that fusing multiple predictions leads to an improvement even in the 2D semantic labelling over baseline single frame predictions. We also show that for a smaller reconstruction dataset with larger variation in prediction viewpoint, the improvement over single frame segmentation increases. Our system is efficient enough to allow real-time interactive use at frame-rates of ≈25Hz.

Context‐Aware Mixed Reality: A Learning‐Based Framework for Semantic‐Level Interaction

“…We solve this issue by utilizing the dense pixel‐wise semantic probability distribution produced by the neural network over every class. Therefore, we can improve the fusion accuracy by projecting the labels with a statistical approach using the Bayesian fusion [ASZ*16] [HFL14] [ZSS17] [MHDL17]. Bayesian fusion enables us to update the label prediction on 2D images within the common coordinate frame of the 3D model: …”

“…Combined with SLAM systems, 2D semantic segmentation can be achieved in 3D environments [RA17] [TTLN17] [ZSS17] [MHDL17], a promising future in robotic vision understanding and autonomous driving. Unlike these existing methods that aimed at providing the semantic understanding of the scene for robots, we are focusing our attention on human interactions.…”

Extending Maps with Semantic and Contextual Object Information for Robot Navigation: a Learning-Based Framework Using Visual and Depth Cues

“…There has been a great interest from the computer vision and robotics communities to exploit object-level information since from the perspective of many applications, it is beneficial to explore the awareness that object instances can provide for assistive computer vision [7,8,9], tracking/SLAM [10,11], or place categorization/scene recognition and life-long mapping [12,13].…”

“…For instance, Nascimento et al [30] applied a binary RGB-D descriptor to feed an Adaboost learning method to classify objects in a navigation task. McCormac et al [11] proposed a method for semantic 3D mapping. Their work combined the formulation of Whelan et al [31], an RGB-D based SLAM system for building a dense point cloud of the scene, with an encoder-decoder convolutional network for pixel-wise semantic segmentation.…”

30‐4: Semantic Simultaneous Localization and Mapping for Augmented Reality

“…The Dense Planar SLAM system [4] seamlessly maps an environment using planar and non-planar regions while tracking the sensor pose in real-time. SemanticFusion [5] combines Convolutional Neural Networks (CNNs) and a state-of-the-art dense SLAM system, ElasticFusion to create a dense semantic map. All of the four semantic SLAM systems mentioned above are focusing on the dense map, though semantic SLAM on the sparse map is fairly enough for some applications in fact.…”