Transfer of Learning from Vision to Touch: A Hybrid Deep Convolutional Neural Network for Visuo-Tactile 3D Object Recognition

Abstract

Transfer of learning or leveraging a pre-trained network and fine-tuning it to perform new tasks has been successfully applied in a variety of machine intelligence fields, including computer vision, natural language processing and audio/speech recognition. Drawing inspiration from neuroscience research that suggests that both visual and tactile stimuli rouse similar neural networks in the human brain, in this work, we explore the idea of transferring learning from vision to touch in the context of 3D object recognition. In particular, deep convolutional neural networks (CNN) pre-trained on visual images are adapted and evaluated for the classification of tactile data sets. To do so, we ran experiments with five different pre-trained CNN architectures and on five different datasets acquired with different technologies of tactile sensors including BathTip, Gelsight, force-sensing resistor (FSR) array, a high-resolution virtual FSR sensor, and tactile sensors on the Barrett robotic hand. The results obtained confirm the transferability of learning from vision to touch to interpret 3D models. Due to its higher resolution, tactile data from optical tactile sensors was demonstrated to achieve higher classification rates based on visual features compared to other technologies relying on pressure measurements. Further analysis of the weight updates in the convolutional layer is performed to measure the similarity between visual and tactile features for each technology of tactile sensing. Comparing the weight updates in different convolutional layers suggests that by updating a few convolutional layers of a pre-trained CNN on visual data, it can be efficiently used to classify tactile data. Accordingly, we propose a hybrid architecture performing both visual and tactile 3D object recognition with a MobileNetV2 backbone. MobileNetV2 is chosen due to its smaller size and thus its capability to be implemented on mobile devices, such that the network can classify both visual and tactile data. An accuracy of 100% for visual and 77.63% for tactile data are achieved by the proposed architecture.

Object Recognition for Humanoid Robots Using Full Hand Tactile Sensor

“…The related work in object recognition by handling is quite diverse due to developments in several aspects of this subject, beginning with studies of human haptic behavior [7] from physiological and psychological studies to understand human haptic perception and unfamiliar objects surveys for recognition [8]. Other studies include humanoid robot hand system design to enable object handling [3], [4], [9], [10], control of the robot hand system of a humanoid robot [11], [12], tactile sensor array design for humanoids robot hand [5], [6], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32]. Methods for recognizing objects based on tactile image recognition [6], [15], [18], [20], [21], [22], [23], [24], [28], [29], [31], [32], [33], [34], [35],…”

“…Other studies include humanoid robot hand system design to enable object handling [3], [4], [9], [10], control of the robot hand system of a humanoid robot [11], [12], tactile sensor array design for humanoids robot hand [5], [6], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32]. Methods for recognizing objects based on tactile image recognition [6], [15], [18], [20], [21], [22], [23], [24], [28], [29], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [4...…”

Review of Bioinspired Vision-Tactile Fusion Perception (VTFP): From Humans to Humanoids

“…Bednarek et al [145] conducted grasp classification experiments on the BiGS dataset to compare the performance of four multimodal fusion algorithms of late fusion, MoE, intermediate fusion and LMF. Rouhafzay et al [146] retrained the convolutional neural network on the successful cases of the BiGS dataset and proposed a hybrid framework MobileNetV2. Results proved that their pretrained deep convolutional neural network on visual images could be effectively transferred to the tactile dataset for classification tasks.…”

“…The dataset was established to first fuse and share the visual and tactile characteristics of different fabrics and then to improve the accuracy of fabric texture recognition tasks. Rouhafzay et al [146] selected 12 kinds of tactile data from the ViTac dataset to retrain and fine-tune their pretrained deep convolutional neural network to ensure the quality of transfer learning. Lee et al [147] proposed a cross-modal sensory data generating framework using a conditional generative adversarial network to generate pseudovisual data from tactile data or to generate pseudotactile data from visual data.…”

Tactile and Vision Perception for Intelligent Humanoids

“…By establishing cross‐mode association between vision and tactile measurement, the knowledge obtained from one modality can be transferred to the other, consequently improving or extending the sensory capabilities. [ 154,155 ]…”