Estimating the Orientation of Objects from Tactile Sensing Data Using Machine Learning Methods and Visual Frames of Reference

Abstract

Underactuated hands are useful tools for robotic in-hand manipulation tasks due to their capability to seamlessly adapt to unknown objects. To enable robots using such hands to achieve and maintain stable grasping conditions even under external disturbances while keeping track of an in-hand object’s state requires learning object-tactile sensing data relationships. The human somatosensory system combines visual and tactile sensing information in their “What and Where” subsystem to achieve high levels of manipulation skills. The present paper proposes an approach for estimating the pose of in-hand objects combining tactile sensing data and visual frames of reference like the human “What and Where” subsystem. The system proposed here uses machine learning methods to estimate the orientation of in-hand objects from the data gathered by tactile sensors mounted on the phalanges of underactuated fingers. While tactile sensing provides local information about objects during in-hand manipulation, a vision system generates egocentric and allocentric frames of reference. A dual fuzzy logic controller was developed to achieve and sustain stable grasping conditions autonomously while forces were applied to in-hand objects to expose the system to different object configurations. Two sets of experiments were used to explore the system capabilities. On the first set, external forces changed the orientation of objects while the fuzzy controller kept objects in-hand for tactile and visual data collection for five machine learning estimators. Among these estimators, the ridge regressor achieved an average mean squared error of 0.077 ∘ . On the second set of experiments, one of the underactuated fingers performed open-loop object rotations and data recorded were supplied to the same set of estimators. In this scenario, the Multilayer perceptron (MLP) neural network achieved the lowest mean squared error of 0.067 ∘ .

Human Somatosensory Processing and Artificial Somatosensation

“…adapted the "what and Where" systems to haptic sensors [81]. The "what" system was used to recognize surface features of objects through haptic sensors, whereas the "where" system provided a description of the contact location on the skin.…”

Tactile Sensors for Robotic Applications

“…The authors of [4] propose the fusion of tactile data and vision data for estimating the pose of in-hand objects grasped with an underactuated robotic hand. These kind of data are very often combined with the aim to reproduce the human approach in manipulation.…”

Generation of Tactile Data From 3D Vision and Target Robotic Grasps

“…Tactile data are used for various tasks [2], like detecting the contours of an object [3], estimating its motion within the robotic hand [5] or calculating its orientation while grasped [4]. Lately, some authors have begun exploring the localisation and reconstruction of the pose of a target object using the tactile sense only [6], [7], just as a human would do when searching for an object in a box.…”