Visual Tracking of Deformation and Classification of Non-Rigid Objects with Robot Hand Probing

Abstract

Performing tasks with a robot hand often requires a complete knowledge of the manipulated object, including its properties (shape, rigidity, surface texture) and its location in the environment, in order to ensure safe and efficient manipulation. While well-established procedures exist for the manipulation of rigid objects, as well as several approaches for the manipulation of linear or planar deformable objects such as ropes or fabric, research addressing the characterization of deformable objects occupying a volume remains relatively limited. The paper proposes an approach for tracking the deformation of non-rigid objects under robot hand manipulation using RGB-D data. The purpose is to automatically classify deformable objects as rigid, elastic, plastic, or elasto-plastic, based on the material they are made of, and to support recognition of the category of such objects through a robotic probing process in order to enhance manipulation capabilities. The proposed approach combines advantageously classical color and depth image processing techniques and proposes a novel combination of the fast level set method with a log-polar mapping of the visual data to robustly detect and track the contour of a deformable object in a RGB-D data stream. Dynamic time warping is employed to characterize the object properties independently from the varying length of the tracked contour as the object deforms. The proposed solution achieves a classification rate over all categories of material of up to 98.3%. When integrated in the control loop of a robot hand, it can contribute to ensure stable grasp, and safe manipulation capability that will preserve the physical integrity of the object.

Multi-Modal Sensing and Robotic Manipulation of Non-Rigid Objects: A Survey

“…The system used a geometric scheme that simulated the physical object behavior based on a position-based dynamics (PBD) model. Similarly, Hui et al [6] proposed a 2D deformation tracking system and applied it to robotic hand manipulation while also adding capabilities for material classification. They used a three-finger Barrett Hand robot and a Kinect sensor to acquire visual and depth data.…”

“…Fugl et al [9] Frank et al [25] Leizea et al [10] Petit et al [26] Lin et al [11] Güler et al [5] Caccamo et al [29] Hui et al [6] Cretu et al [22] Arriola-Rios and Wyatt [21] Drimus et al [14] Mira et al [16] Delgado et al [18] Navarro-Alarcon et al [12] Alambeigi et al [13]…”

“…Alternatively, Hui et al [6,41] used an implicit geometric representation named fast level set for tracking the deformable surface. Initially, the object was segmented from the raw colour and depth data by applying RANSAC and k-d tree search algorithms to remove the undesired surfaces.…”

“…Despite this, they are still in an early stage and only simple applications have been integrated. Güler et al [5] Gil et al [28] Mateo et al [39] Hui et al [6] Cretu et al [22] Tawbe and Cretu [43] Arriola-Rios and Wyatt [21] Cherubini et al [45] Bodenhagen et al [46] Güler et al [47] Mira et al [16] Delgado et al [18] Navarro-Alarcon et al [12] Alambeigi et al [13] 1 approaches that generalize the type of deformation.…”

Reactive motion planning for robotic manipulators and interaction control with deformable environment

“…Recent papers involve a robotic system into deformation tracking tasks. More specifically, in [137,138] the authors characterize generic deformable objects by visually observing their interaction with a robotic hand and in [139] surface variations caused by the contact of a simulated robotic hand are measured visually on Point Cloud data. Tian and Jia in [140] extend their work on shape modelling of shell-like objects, which are grasped by a robotic hand.…”