Building a Library of Tactile Skills Based on FingerVision

Abstract

Camera-based tactile sensors are emerging as a promising inexpensive solution for tactile-enhanced manipulation tasks. A recently introduced FingerVision sensor was shown capable of generating reliable signals for force estimation, object pose estimation, and slip detection. In this paper, we build upon the FingerVision design, improving already existing control algorithms, and, more importantly, expanding its range of applicability to more challenging tasks by utilizing raw skin deformation data for control. In contrast to previous approaches that rely on the average deformation of the whole sensor surface, we directly employ local deviations of each spherical marker immersed in the silicone body of the sensor for feedback control and as input to learning tasks. We show that with such input, substances of varying texture and viscosity can be distinguished on the basis of tactile sensations evoked while stirring them. As another application, we learn a mapping between skin deformation and force applied to an object. To demonstrate the full range of capabilities of the proposed controllers, we deploy them in a challenging architectural assembly task that involves inserting a load-bearing element underneath a bendable plate at the point of maximum load.

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2-D to 3-D: A Review

“…The above work was mainly completed by analyzing tactile behaviors for grasping and manipulation tasks, and designing appropriate control strategies [22]. Based on FingerVision, Belousov et al developed a controller library that contained rich control strategies and tactile skills [87], and completed two challenging tasks: distinguishing objects with different characteristics and architectural assembly.…”

“… Optical flow method [28], [37]  Finite element mode [63], [44]  Neural network [58], [53] mainly using the machine learningbased approaches  Speckle detection [77], [80]  Feature enhancement [81], [82] mainly using the physical model-based approaches  Stereo vision [25], [94]  Virtual stereo vision [23], [26] mainly using the physical model-based approaches  Contact area [24]  2D force distribution [17]  Slip field [79]  Contact area [20]  2D force distribution [86]  Slip field [21]  Contact area [91]  Friction coefficient [23]  2D force distribution [97] 3D tactile perception  Geometric features [46]  3D geometry [18]  3D force distribution [63]  Geometric features [82]  3D geometry [82]  3D force distribution [87]  Geometric features [ commonly used in the field of visuotactile sensing. By using a camera to photograph the marks prepared on the sensor contact elastomer, a tactile image containing the position change of the markers can be obtained, and the tactile information can be further obtained by post-processing and analyzing the tactile image.…”

Marker Displacement Method Used in Vision-Based Tactile Sensors—From 2D to 3D: A Review

“… Optical flow method [28], [37]  Finite element mode [63], [44]  Neural network [58], [53] mainly using the machine learningbased approaches  Speckle detection [77], [80]  Feature enhancement [81], [82] mainly using the physical model-based approaches  Stereo vision [25], [94]  Virtual stereo vision [23], [26] mainly using the physical model-based approaches  Contact area [24]  2D force distribution [17]  Slip field [79]  Contact area [20]  2D force distribution [86]  Slip field [21]  Contact area [91]  Friction coefficient [23]  2D force distribution [97] 3D tactile perception  Geometric features [46]  3D geometry [18]  3D force distribution [63]  Geometric features [82]  3D geometry [82]  3D force distribution [87]  Geometric features [ commonly used in the field of visuotactile sensing. By using a camera to photograph the marks prepared on the sensor contact elastomer, a tactile image containing the position change of the markers can be obtained, and the tactile information can be further obtained by post-processing and analyzing the tactile image.…”

PyTouch: A Machine Learning Library for Touch Processing

“…Previous attempts include studying friction models [10], physical property understanding [11], [12], [13], in-hand manipulation, determining physical manipulation characteristics and grasp success. [14] demonstrates individual capabilities related to tactile touch which allow for robotic learning in providing given forces to an object, determining density and texture through stirring actions, and various types of in hand or arm rotation of objects. Other work such as [15] learns robotic grasp success through an end-to-end action-conditional model based off of raw tactile sensor input data.…”