Spatio-temporal Attention Model for Tactile Texture Recognition

Cao, Guanqun; Zhou, Yi; Bollegala, Danushka; Luo, Shan

doi:10.1109/iros45743.2020.9341333

Cited by 36 publications

(14 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, few works have explored how Transformer models can benefit robot learning, which involves more dynamical interactions compared with common image learning tasks. In [24] and [25], they combined the Transformer encoders with CNN networks. Specifically, they first used CNN networks to extract per-frame features and then processed these features via the Transformer encoders.…”

Section: Transformersmentioning

confidence: 99%

Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

Han¹,

Batra²,

Boyd³

et al. 2021

Preprint

View full text Add to dashboard Cite

Reliable robotic grasping, especially with deformable objects such as fruits, remains a challenging task due to underactuated contact interactions with a gripper, unknown object dynamics and geometries. In this study, we propose a Transformer-based robotic grasping framework for rigid grippers that leverage tactile and visual information for safe object grasping. Specifically, the Transformer models learn physical feature embeddings with sensor feedback through performing two pre-defined explorative actions (pinching and sliding) and predict a grasping outcome through a multilayer perceptron (MLP) with a given grasping strength. Using these predictions, the gripper predicts a safe grasping strength via inference. Compared with convolutional recurrent networks (CNN), the Transformer models can capture the long-term dependencies across the image sequences and process spatial-temporal features simultaneously. We first benchmark the Transformer models on a public dataset for slip detection. Following that, we show that the Transformer models outperform a CNN+LSTM model in terms of grasping accuracy and computational efficiency. We also collect our fruit grasping dataset and conduct online grasping experiments using the proposed framework for both seen and unseen fruits. Our codes and dataset are public on GitHub 1 .

show abstract

Section: Transformersmentioning

confidence: 99%

Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

Han¹,

Batra²,

Boyd³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Due to the soft opaque tactile membrane, the captured images are robust to external light variations, and capture information of the touched surface's geometry structure, unlike most conventional tactile sensors that measure the touching force. Leveraging the high resolution of the captured tactile images, high accuracy geometry reconstructions are produced in [31][32][33][34][35][36]. In [31], this sensor was used as fingers of a robotic gripper to insert a USB cable into the correspondent port effectively.…”

Section: Image-based Optical Tactile Sensorsmentioning

confidence: 99%

GelTip Tactile Sensor for Dexterous Manipulation in Clutter

Gomes¹,

Si²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Tactile sensing is an essential capability for robots that carry out dexterous manipulation tasks. While cameras, Lidars and other remote sensors can assess a scene globally and instantly, tactile sensors can reduce their measurement uncertainties and gain information about the local physical interactions between the in-contact objects and the robot, that are often not accessible via remote sensing. Tactile sensors can be grouped into two main categories: electronic tactile skins and camera based optical tactile sensors. The former are slim and can be fitted to different body parts, whereas the latter assume a more prismatic shape and have much higher sensing resolutions, offering a good advantage for being used as robotic fingers or fingertips. One of such optical tactile sensors is our GelTip sensor that is shaped as a finger and can sense contacts on any location of its surface. As such, the GelTip sensor is able to detect contacts from all the directions, like a human finger. To capture these contacts, it uses a camera installed at its base to track the deformations of the opaque elastomer that covers its hollow, rigid and transparent body. Thanks to this design, a gripper equipped with GelTip sensors is capable of simultaneously monitoring contacts happening inside and outside its grasp closure. Experiments carried out using this sensor demonstrate how contacts can be localised, and more importantly, the advantages, and even possibly a necessity, of leveraging all-around touch sensing in dexterous manipulation tasks in clutter where contacts may happen at any location of the finger. In particular, experiments carried out in a Blocks World environment show that the detected contacts on the fingers can be used to adapt planned actions during the different moments of the reach-to-grasp motion. All the materials for the fabrication of the GelTip sensor can be found at https://danfergo.github.io/geltip/

show abstract

“…Recently, a spatio-temporal attention model (STAM) is proposed for the tactile texture recognition [49]. The textures are shown in Fig.…”

Section: Spatio-temporal Attention Modelmentioning

confidence: 99%

Multimodal perception for dexterous manipulation

Cao¹,

Si²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Humans usually perceive the world in a multimodal way that vision, touch, sound are utilised to understand surroundings from various dimensions. These senses are combined together to achieve a synergistic effect where the learning is more effectively than using each sense separately. For robotics, vision and touch are two key senses for the dexterous manipulation. Vision usually gives us apparent features like shape, color, and the touch provides local information such as friction, texture, etc. Due to the complementary properties between visual and tactile senses, it is desirable for us to combine vision and touch for a synergistic perception and manipulation. Many researches have been investigated about multimodal perception such as cross-modal learning, 3D reconstruction, multimodal translation with vision and touch. Specifically, we propose a cross-modal sensory data generation framework for the translation between vision and touch, which is able to generate realistic pseudo data. By using this cross-modal translation method, it is desirable for us to make up inaccessible data, helping us to learn the object's properties from different views. Recently, the attention mechanism becomes a popular method either in visual perception or in tactile perception. We propose a spatio-temporal attention model for tactile texture recognition, which takes both spatial features and time dimension into consideration. Our proposed method not only pays attention to the salient features in each spatial feature, but also models the temporal correlation in the through the time. The obvious improvement proves the efficiency of our selective attention mechanism. The spatio-temporal attention method has potential in many applications such as grasping, recognition, and multimodal perception.

show abstract

Spatio-temporal Attention Model for Tactile Texture Recognition

Cited by 36 publications

References 24 publications

Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

GelTip Tactile Sensor for Dexterous Manipulation in Clutter

Multimodal perception for dexterous manipulation

Contact Info

Product

Resources

About