Dexterous object manipulation by a multi-fingered robotic hand with visual-tactile fingertip sensors

Abstract

In this paper, a novel visual-tactile sensor is proposed; additionally, an object manipulation method for a multi-fingered robotic hand grasping an object is proposed by detecting a contact position using the visual-tactile sensor. The visual-tactile sensor is composed of a hemispheric fingertip made of soft silicone with a hollow interior and a general USB camera located inside the fingertip to detect the displacement of the many point markers embedded in the silicone. The deformation of each point marker due to a contact force is measured, and a contact position is estimated reliably through a novel method of creating virtual points to determine the point clouds. The aim is to demonstrate both the estimation performance of the new visual-tactile sensor and its usefulness in a grasping and manipulation task. By using the contact position obtained from the proposed sensor and the position of each fingertip obtained from kinematics, the position and orientation of a grasped object are estimated and controlled. The effectiveness of the method is illustrated through numerical simulation and its practical use is demonstrated through grasping and manipulating experiments.

Deformable Object Manipulation in Caregiving Scenarios: A Review

“…Another example can be found in the work of Choi et al [106], who developed a multifingered robotic system for dexterously manipulating deformable objects using visual and tactile sensing. Their approach used a combination of visual features extracted from RGB-D images and tactile measurements from an array of pressure sensors to model the object's deformation.…”

“…The significance of planning and control in deformable object manipulation within caregiving scenarios cannot be overstated [72,105,106]. In these contexts, robots and assistive systems must interact with delicate and often unpredictable objects, such as human tissues, textiles, or prosthetics, making precise planning and control crucial for patient safety and comfort [32].…”

Leveraging Tactile Sensors for Low Latency Embedded Smart Hands for Prosthetic and Robotic Applications

“…State-of-the-art sensor solutions, from academia and industry, are typically hard to manufacture, expensive, and lacking in temporal as well as spatial resolution [17]- [19]. The majority of the solutions mainly focus on replicating the fingers tips [3], [10], [20], [21] or have sensors covering only a limited part of the hand palm [1]- [3], [22]- [24]. While [25] presents a system with full hand coverage, they use a small number of sensing elements (i.e., 16) and therefore only prove the functionality of the system with seven objects.…”