Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

Han, Yunhai; Batra, Rahul; Boyd, Nathan S.; Zhao, Tao; She, Yu; Hutchinson, Seth; Zhao, Yupei

doi:10.48550/arxiv.2112.06374

Cited by 3 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, Low [27] used you only look once, version 3 (YOLOv3) for object detection and successfully switched between three gripper poses based on detection results. Other networks, including convolutional neural network (CNN) [183] and Transformer, [184] have also provided reliable solutions for object detection, thus enhancing grasping capabilities. The performance of sensors in the aforementioned four categories is documented in Table 7-10, along with their respective strengths and limitations.…”

Section: Imaging Sensorsmentioning

confidence: 99%

Intelligent Soft Robotic Grippers for Agricultural and Food Product Handling: A Brief Review with a Focus on Design and Control

Liu,

Hou,

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Advances in material sciences, control algorithms, and manufacturing techniques have facilitated rapid progress in soft grippers, propelling their adoption in various fields. In this review article, a comprehensive overview of the design and control aspects of intelligent soft robotic grippers tailored specifically for agricultural product handling is provided. Soft grippers have emerged as a promising solution for handling delicate and fragile objects. In this article, the recent progress in various gripper design, including fluidic and mechanical grippers, is elucidated and the role of advanced control approaches in enabling intelligent functions, such as object classification and grasping condition evaluation, is explored. Moreover, the challenges and opportunities pertaining to implementation of soft grippers in the agricultural industry are thoroughly discussed. While most demonstrations of soft grippers and their control strategies remain at the experimental stage, in this article, it is aimed to provide insights into the potential applications of soft grippers in agricultural product handling, thereby inspiring future research in this field.

show abstract

Section: Imaging Sensorsmentioning

confidence: 99%

Intelligent Soft Robotic Grippers for Agricultural and Food Product Handling: A Brief Review with a Focus on Design and Control

Liu,

Hou,

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

show abstract

“…Yi et al [22] extracted tactile features from signals and proposed a new genetic algorithm-based integrated hybrid sparse limit learning machine for grasp stability recognition tasks. Han et al [23] introduced a Transformer-based robotic grasping framework for rigid gripper robots, leveraging tactile and visual information to ensure secure object grasping. Evaluation on slip detection and fruit grasping datasets demonstrated that Transformer models exhibit higher grasping accuracy and computational efficiency compared to traditional CNN + LSTM models.…”

Section: Target Grasping Status Detectionmentioning

confidence: 99%

RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets

Ruan,

Zhu,

Zhao

et al. 2023

Mathematics

View full text Add to dashboard Cite

Accurate grasping state detection is critical to the dexterous operation of robots. Robots must use multiple modalities to perceive external information, similar to humans. The direct fusion method of visual and tactile sensing may not provide effective visual–tactile features for the grasping state detection network of the target. To address this issue, we present a novel visual–tactile fusion model (i.e., RFCT) and provide an incremental dimensional tensor product method for detecting grasping states of weak-stiffness targets. We investigate whether convolutional block attention mechanisms (CBAM) can enhance feature representations by selectively attending to salient visual and tactile cues while suppressing less important information and eliminating redundant information for the initial fusion. We conducted 2250 grasping experiments using 15 weak-stiffness targets. We used 12 targets for training and three for testing. When evaluated on untrained targets, our RFCT model achieved a precision of 82.89%, a recall rate of 82.07%, and an F1 score of 81.65%. We compared RFCT model performance with various combinations of Resnet50 + LSTM and C3D models commonly used in grasping state detection. The experimental results show that our RFCT model significantly outperforms these models. Our proposed method provides accurate grasping state detection and has the potential to provide robust support for robot grasping operations in real-world applications.

show abstract

Vahagn: VisuAl Haptic Attention Gate Net for slip detection

Wang,

Ji,

Yang

2024

Front. Neurorobot.

View full text Add to dashboard Cite

IntroductionSlip detection is crucial for achieving stable grasping and subsequent operational tasks. A grasp action is a continuous process that requires information from multiple sources. The success of a specific grasping maneuver is contingent upon the confluence of two factors: the spatial accuracy of the contact and the stability of the continuous process.MethodsIn this paper, for the task of perceiving grasping results using visual-haptic information, we propose a new method for slip detection, which synergizes visual and haptic information from spatial-temporal dual dimensions. Specifically, the method takes as input a sequence of visual images from a first-person perspective and a sequence of haptic images from a gripper. Then, it extracts time-dependent features of the whole process and spatial features matching the importance of different parts with different attention mechanisms. Inspired by neurological studies, during the information fusion process, we adjusted temporal and spatial information from vision and haptic through a combination of two-step fusion and gate units.Results and discussionTo validate the effectiveness of method, we compared it with traditional CNN net and models with attention. It is anticipated that our method achieves a classification accuracy of 93.59%, which is higher than that of previous works. Attention visualization is further presented to support the validity.

show abstract

Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

Cited by 3 publications

References 17 publications

Intelligent Soft Robotic Grippers for Agricultural and Food Product Handling: A Brief Review with a Focus on Design and Control

Intelligent Soft Robotic Grippers for Agricultural and Food Product Handling: A Brief Review with a Focus on Design and Control

RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets

Vahagn: VisuAl Haptic Attention Gate Net for slip detection

Contact Info

Product

Resources

About