Manipulator and object tracking for in-hand 3D object modeling

Abstract

Recognizing and manipulating objects is an important task for mobile robots performing useful services in everyday environments. While existing techniques for object recognition related to manipulation provide very good results even for noisy and incomplete data, they are typically trained using data generated in an offline process. As a result, they do not enable a robot to acquire new object models as it operates in an environment. In this paper we develop an approach to building 3D models of unknown objects based on a depth camera observing the robot's hand while moving an object. The approach integrates both shape and appearance information into an articulated Iterative Closest Point approach to track the robot's manipulator and the object. Objects are modeled by sets of surfels, which are small patches providing occlusion and appearance information. Experiments show that our approach provides very good 3D models even when the object is highly symmetric and lacks visual features and the manipulator motion is noisy. Autonomous object modeling represents a step toward improved semantic understanding, which will eventually enable robots to reason about their environments in terms of objects and their relations rather than through raw sensor data.

Efficient generation of 3D surfel maps using RGB–D sensors

“…The proposed approach to surfel mapping follows to some extent solutions described by Krainin et al (2010) and Henry et al (2012); however, it contains some novel elements that provide an efficiency improvement. The first improvement is the acceleration of the surfel map update procedure thanks to "octree-based frustum culling".…”

“…The measure of confidence may be defined as the number of views where the surfel was observed, but also (in a more complex solution) as a histogram of viewing directions. Surfels became useful as building blocks of 3D maps, as proposed by Weise et al (2009) and followed by others, e.g., Krainin et al (2010) or Henry et al (2012), mainly due to point compression and noise reduction capabilities of surfel-based mapping algorithms.…”

“…In the model update procedure, the method proposed by Weise et al (2009) and Krainin et al (2010) is followed quite closely. For each subsequent j-th sensor scan from visual odometry, the j-th sensor pose G Cj T with respect to the global reference frame G is obtained.…”

“…"voxel" representation-the 3D space is divided into small "cubes" (Marder-Eppstein et al, 2010), 2. "surfel" representation-the space is modelled by an arrangement of surface elements (Krainin et al, 2010), 3. MLS (multi-level surface)-contains a list of obstacles (occupied voxels) associated with each element of a 2D grid (Triebel et al, 2006), 4. octal trees (octrees)-constitute an effective implementation of the voxel map using the octal tree structure (Wurm et al, 2010), 5. combined voxel-surfel representation (e.g., the multi-resolution surfel map).…”

Xtru3D: Single-View 3D Object Reconstruction from Color and Depth Data

“…Moreover, for manipulation applications, once the object is grasped, several approaches for model re nement have been considered: on the one hand, the integration of single view estimation with the incremental model re nements techniques of e.g. [23] and [24] would be interesting. On the other hand, focusing on recent advances on sensor fusion for manipulation of unknown objects, this model re nement could be developed fusing visual and tactile data [25].…”