Incremental Object Database: Building 3D Models from Multiple Partial Observations

Abstract

Collecting 3D object datasets involves a large amount of manual work and is time consuming. Getting complete models of objects either requires a 3D scanner that covers all the surfaces of an object or one needs to rotate it to completely observe it. We present a system that incrementally builds a database of objects as a mobile agent traverses a scene. Our approach requires no prior knowledge of the shapes present in the scene. Object-like segments are extracted from a global segmentation map, which is built online using the input of segmented RGB-D images. These segments are stored in a database, matched among each other, and merged with other previously observed instances. This allows us to create and improve object models on the fly and to use these merged models to reconstruct also unobserved parts of the scene. The database contains each (potentially merged) object model only once, together with a set of poses where it was observed. We evaluate our pipeline with one public dataset, and on a newly created Google Tango dataset containing four indoor scenes with some of the objects appearing multiple times, both within and across scenes.

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

“…SLAM++ [18] builds object-oriented maps by detecting recognized elements in RGB-D data, but is limited to work with a database of objects for which exact geometric models need to be known in advance. A number of other works have addressed the task of detecting and segmenting individual semantically meaningful objects in 3D scenes without predefined shape templates [3]- [7], [9]. Recent learning-based approaches segment individual instances of semantically annotated objects in reconstructed scenes with little or no prior information about their exact appearance while at the same time handling substantial intra-class variability [3]- [6].…”

“…In contrast, purely geometry-based methods operate under open-set conditions and are able to discover novel, previously unobserved objects in the scene [7], [9]. The work in [9] provides a complete and exhaustive geometric segmentation of the scene.…”

“…the robot will inevitably encounter novel objects of previously unseen categories. Monitor Keyboard Suitcase Table Chair Mouse Refrigerator Plant Backpack Cup Microwave Unknown As opposed to a geometry-only segmentation from our previous work [7] shown in Figure (d), the proposed framework prevents over-segmentation of recognized articulated objects and segments them as one instance despite their non-convex shape (blue circle), assigning each a semantic category shown in Figure (c). At the same time, the proposed approach discovers novel, previously unseen object-like elements of unknown class (red circle).…”

“…In particular, the proposed object-oriented mapping framework retrieves the pose and shape of recognized semantic objects, as well as of newly discovered, previously unobserved object-like instances. The proposed system builds on top of the incremental geometry-based scene segmentation approach from our previous work in [7] and extends it to produce a complete instance-aware semantic mapping framework. Figure 1 shows the object-centric map of an office scene reconstructed with the proposed approach.…”

VersaVIS—An Open Versatile Multi-Camera Visual-Inertial Sensor Suite

“…Fig. 12: Object segmentation and reconstruction based on [21] and [22] using data from the RGB-D-I sensor and camera poses obtained with maplab [18]. that is part of the map.…”