Vision-Based Tactile Sensor Mechanism for the Estimation of Contact Position and Force Distribution Using Deep Learning

Abstract

This work describes the development of a vision-based tactile sensor system that utilizes the image-based information of the tactile sensor in conjunction with input loads at various motions to train the neural network for the estimation of tactile contact position, area, and force distribution. The current study also addresses pragmatic aspects, such as choice of the thickness and materials for the tactile fingertips and surface tendency, etc. The overall vision-based tactile sensor equipment interacts with an actuating motion controller, force gauge, and control PC (personal computer) with a LabVIEW software on it. The image acquisition was carried out using a compact stereo camera setup mounted inside the elastic body to observe and measure the amount of deformation by the motion and input load. The vision-based tactile sensor test bench was employed to collect the output contact position, angle, and force distribution caused by various randomly considered input loads for motion in X, Y, Z directions and RxRy rotational motion. The retrieved image information, contact position, area, and force distribution from different input loads with specified 3D position and angle are utilized for deep learning. A convolutional neural network VGG-16 classification modelhas been modified to a regression network model and transfer learning was applied to suit the regression task of estimating contact position and force distribution. Several experiments were carried out using thick and thin sized tactile sensors with various shapes, such as circle, square, hexagon, for better validation of the predicted contact position, contact area, and force distribution.

Automated Inorganic Pigment Classification in Plastic Material Using Terahertz Spectroscopy

“…The CNN algorithm is a subset of deep neural networks and deep learning paradigms [ 22 ], and has proven its effectiveness as an image, speech recognition, face detection, futures extraction algorithm. The novel research confirms that CNNs have advantages in series forecasting, a data-driven approach for diagnostic and fault classification of various industrial processes and applications [ 23 , 24 , 25 , 26 , 27 ].…”

Speckle-based high-resolution multimodal soft sensing

“…Owing to the high sensitivity realized with the optical scattering interference, the proposed approach can enable the sensing of physical contacts with high spatial resolution of few tens of micrometers. Thus, it outperforms previous electrical and optical approaches used in this field of study 33 – 36 , although the precision is slightly lower than that of the state-of-the-art measurements, such as speckle interferometry, holography, and projection techniques 42 , 44 , 47 , 48 . Higher spatial resolution and greater precision of measurement of the indentation can be achieved by using a larger number of training samples.…”

“…A previous approach to achieve high spatial sensing capability was based on the integration of a large number of sensors to form a sensor matrix with numerous wire connections; however, this usually incurs high integration complexity. Although a vision-based tactile sensing approach using a marker displacement is easy to manufacture 36 , the spatial resolution is limited, and it is difficult to simultaneously detect other parameters such as temperature.…”

Optical Fiber Array Sensor for Force Estimation and Localization in TAVI Procedure: Design, Modeling, Analysis and Validation

“…In such a configuration, all the fibers will exhibit levels of intensity loss. With proper non-linear calibration, i.e., learning-based rate-dependent methods [ 38 ], a mapping between states of the intensity loss in the four fibers and the position and magnitude of the external force can be obtained. Since in this experiment only one fiber was utilized, obtaining such calibration was not possible.…”