3D object segmentation for shelf bin picking by humanoid with deep learning and occupancy voxel grid map

Abstract

Picking objects in a narrow space such as shelf bins is an important task for humanoid to extract target object from environment. In those situations, however, there are many occlusions between the camera and objects, and this makes it difficult to segment the target object three dimensionally because of the lack of three dimentional sensor inputs. We address this problem with accumulating segmentation result with multiple camera angles, and generating voxel model of the target object. Our approach consists of two components: first is object probability prediction for input image with convolutional networks, and second is generating voxel grid map which is designed for object segmentation. We evaluated the method with the picking task experiment for target objects in narrow shelf bins. Our method generates dense 3D object segments even with occlusions, and the real robot successfully picked target objects from the narrow space.

Probabilistic 3D multilabel real-time mapping for multi-object manipulation

“…The proposed method extends our previous method for a single label of objects [1], and represents object-label and collision with multilabel occupancies in each voxel. For real-time map generation, we extend octomap [3] for multilabel objects, which is firstly proposed to efficiently map single-label, collision object, occupancy.…”

“…In addition to these works on 2D segmentation, three-dimentional segmentation is required for robot to conduct tasks in the real world. In order to achieve this, previous works propose projection-based approach projecting segmented pixels to 3D points in a single view (2.5D) [9], mapping-based approach with binary object existence [12] and probabilistic existence [1] for a single target object. And as for fully 3D-based approach, model matching is tackled [13] [14] using various 3D features [15] [16].…”

“…And as for fully 3D-based approach, model matching is tackled [13] [14] using various 3D features [15] [16]. We use a mapping-based approach with multiple views extending our previous method [1] for multi-label objects to deal with object occlusions and flexible objects for which static 3D model is less effective. Our method segments multi-label objects in a single multi-view action, and effective to acquire the dense 3D information of objects in an environment with heavy occlusions: selfocclusion and occlusion by others.…”

“…However, their research lacks of consideration of real-time efficiency, which is crucial for 3D segmentation of objects and its manipulation. On the other hand, we have proposed real-time 3D mapping for a single label object [1]. In order for robot to conduct tasks that demands multi-object segmentation at once; for example multi-object manipulation ( Fig.1), we propose a method to construct three-dimensional map for multi-label objects in real-time.…”

“…In this paper, we propose a method to generate three-dimensional map with multilabel occupancy in real-time. Extending our previous work [1] in which only target label occupancy is mapped, we achieve multilabel object segmentation in a single looking around action. We evaluate our method by testing segmentation accuracy with 39 different objects, and applying it to a manipulation task of multiple objects in the experiments.…”