Meaningful maps with object-oriented semantic mapping

Abstract

Fig. 1: We demonstrate object-oriented semantic mapping using RGB-D data that scales from small desktop environments (left) to offices (middle) and whole labs (right). The pictures show 3D map structures with objects colored according to their semantic class. We do not merely project semantic labels for individual 3D points, but rather maintain objects as the central entity of the map, freeing it from the requirement for a-priori 3D object models in [1]. To achieve this, our system creates and extends 3D object models while continuously mapping the environment. Object detection and classification is performed using a Convolutional Network, while an unsupervised 3D segmentation algorithm assigns a segment of 3D points to every object detection. These segmented object detections are then either fused with existing objects, or added as a new object to the map. ORB-SLAM2 provides a global SLAM solution that enables us to reconstruct a 3D model of the environment that contains both non-object structure and objects of various types.Abstract-For intelligent robots to interact in meaningful ways with their environment, they must understand both the geometric and semantic properties of the scene surrounding them. The majority of research to date has addressed these mapping challenges separately, focusing on either geometric or semantic mapping. In this paper we address the problem of building environmental maps that include both semantically meaningful, object-level entities and point-or mesh-based geometrical representations. We simultaneously build geometric point cloud models of previously unseen instances of known object classes and create a map that contains these object models as central entities. Our system leverages sparse, feature-based RGB-D SLAM, image-based deep-learning object detection and 3D unsupervised segmentation.

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

“…SLAM++ [18] builds object-oriented maps by detecting recognized elements in RGB-D data, but is limited to work with a database of objects for which exact geometric models need to be known in advance. A number of other works have addressed the task of detecting and segmenting individual semantically meaningful objects in 3D scenes without predefined shape templates [3]- [7], [9]. Recent learning-based approaches segment individual instances of semantically annotated objects in reconstructed scenes with little or no prior information about their exact appearance while at the same time handling substantial intra-class variability [3]- [6].…”

“…A number of other works have addressed the task of detecting and segmenting individual semantically meaningful objects in 3D scenes without predefined shape templates [3]- [7], [9]. Recent learning-based approaches segment individual instances of semantically annotated objects in reconstructed scenes with little or no prior information about their exact appearance while at the same time handling substantial intra-class variability [3]- [6]. However, by relying on a strong supervisory signal of the predefined classes during training, a purely learning-based segmentation fails to discover novel objects of unknown class in the scene.…”

A clustering based on optimization for object oriented quality prediction

“…Furthermore, the code that implements normal behavior and the code for its exception handlers are tangled together resulting in a loss of code comprehension [8][9][10][11][12]. Aspect-oriented programming, which group's related functionality that crosses modular boundaries, into concerns provides a solution to these problems.…”

SceneCode: Monocular Dense Semantic Reconstruction Using Learned Encoded Scene Representations

“…At the other end of the scale are approaches which explicitly recognise object instances and build scene models as 3D object graphs [27,35,30,31]. These representations have the token-like character we are looking for, but are limited to mapping discrete 'blob-like' objects from known classes and leave large fractions of scenes undescribed.…”