Deep Active Cross-Modal Visuo-Tactile Transfer Learning for Robotic Object Recognition

Abstract

We propose for the first time, a novel deep active visuo-tactile cross-modal full-fledged framework for object recognition by autonomous robotic systems. Our proposed network xAVTNet is actively trained with labelled point clouds from a vision sensor with one robot and tested with an active tactile perception strategy to recognise objects never touched before using another robot. We propose a novel visuo-tactile loss (VTLoss) to minimise the discrepancy between the visual and tactile domains for unsupervised domain adaptation. Our framework leverages the strengths of deep neural networks for cross-modal recognition along with active perception and active learning strategies for increased efficiency by minimising redundant data collection. Our method is extensively evaluated on a real robotic system and compared against baselines and other state-of-art approaches. We demonstrate clear outperformance in recognition accuracy compared to the state-of-art visuo-tactile cross-modal recognition method.

Skin‐Inspired Thermosensitive Tactile Sensor Based on Thermally Conductive and Viscous Interface Composites for Rocks

“…Second, materials with high thermal conductivity contribute to effective heat dissipation, preventing heat accumulation and potential damage to sensor components. [ 40 ] This guarantees the long‐term reliable operation of the sensor. [ 41 ] In conclusion, high‐thermal‐conductivity thermal interface materials enhance the sensitivity, responsiveness, and performance of thermosensitive tactile sensor.…”

Efficient Visuo-Haptic Object Shape Completion for Robot Manipulation

“…However, these are not as effective as an uncertainty-driven approach. Uncertainty can come from the Gaussian distribution [16]- [19], [21]; from the Monte Carlo dropout [24]; or from the Signed Distance Function (SDF) [1], [25]. Alternatively, it can be learned where to touch as in Smith et al [23].…”

Touch if it's Transparent! ACTOR: Active Tactile-Based Category-Level Transparent Object Reconstruction

“…While these works focus on opaque objects, limited works exist for the reconstruction of transparent objects. Recently, deep learning methods have been used for point cloud based shape completion given partial or noisy input point clouds [23,24]. Seminal works on PointNet [25] allowed using raw point clouds as inputs to deep networks for the task of classification and semantic segmentation.…”

“…Using the constructed object shape for pose estimation of a transparent object through tactile sensing brings further challenges due to the nature of the tactile data. Typical poseestimation methods for visual perception perform poorly with tactile data as they are sparse and extracted sequentially through contact probing [1,4,24,[28][29][30]. In summary, there are limitations in the state-of-the-art for the reconstruction and further applications such as pose estimation of transparent objects with tactile perception: (a) existing reconstruction strategies such as GPIS fail to capture fine shape details with sparse tactile input data, (b) directly deploying deep learning based strategies for shape completion with sparse input data is impractical as the collection of a large dataset of tactile data for training is prohibitively expensive, (c) existing tactile-based pose estimation techniques rely upon known object models or shape primitives but category-level tactilebased pose estimation wherein objects without a priori known CAD models but belong to a known category is necessary.…”