Variable In-Hand Manipulations for Tactile-Driven Robot Hand via CNN-LSTM

Funabashi, Satoshi; Ogasa, Shun; Isobe, Tomoki; Ogata, Tetsuya; Schmitz, Alexander; Tomo, Tito Pradhono; Sugano, Shigeki

doi:10.1109/iros45743.2020.9341484

Cited by 10 publications

(7 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CNNs were used as they can process spatially distributed information. We also confirmed that one CNN could produce various in-grasp manipulation motions using labels allocated for each motion [7]. Moreover, combined CNNs were proposed and used for object recognition with a multi-fingered hand [4].…”

Section: Introductionsupporting

confidence: 65%

Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Funabashi

Isobe

Hongyi

et al. 2022

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize it for achieving manipulation skills is still an open issue. This paper presents a control method based on a graph convolutional network (GCN) which extracts geodesical features from the tactile data with complicated sensor alignments. Moreover, object property labels are provided to the GCN to adjust in-hand manipulation motions. Distributed tri-axial tactile sensors are mounted on the fingertips, finger phalanges and palm of an Allegro hand, resulting in 1152 tactile measurements. Training data is collected with a data-glove to transfer human dexterous manipulation directly to the robot hand. The GCN achieved high success rates for in-hand manipulation. We also confirmed that fragile objects were deformed less when correct object labels were provided to the GCN. When visualizing the activation of the GCN with a PCA, we verified that the network acquired geodesical features. Our method achieved stable manipulation even when an experimenter pulled a grasped object and for untrained objects. A Project page including accompanying video and supplementary materials can be found at https://sites.google.com/site/bashifunabashi/multifinger-project/in-hand-manipulation

show abstract

Section: Introductionsupporting

confidence: 65%

Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Funabashi

Isobe

Hongyi

et al. 2022

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…After offline training of the model based on simulation data, the RMSE is 1.21 after the model converges. To verify the superiority of the system model designed in this paper, the model is compared with the MLP model [ 37 ] and CNN-LSTM model [ 38 ], which are commonly used in data-driven system modeling. In the experiment, the MLP model and the regressor of our model adopt the same structure.…”

Section: Methodsmentioning

confidence: 99%

Robotic Manipulation Planning for Automatic Peeling of Glass Substrate Based on Online Learning Model Predictive Path Integral

Hou

Wang

Zou

et al. 2022

Sensors

View full text Add to dashboard Cite

Autonomous planning robotic contact-rich manipulation has long been a challenging problem. Automatic peeling of glass substrates of LCD flat panel displays is a typical contact-rich manipulation task, which requires extremely high safe handling through the manipulation process. To this end of peeling glass substrates automatically, the system model is established from data and is used for the online planning of the robot motion in this paper. A simulation environment is designed to pretrain the process model with deep learning-based neural network structure to avoid expensive and time-consuming collection of real-time data. Then, an online learning algorithm is introduced to tune the pretrained model according to the real-time data from the peeling process experiments to cover the uncertainties of the real process. Finally, an Online Learning Model Predictive Path Integral (OL-MPPI) algorithm is proposed for the optimal trajectory planning of the robot. The performance of our algorithm was validated through glass substrate peeling tasks of experiments.

show abstract

“…For instance, recent work has trained a policy to leverage multimodal feedback in contactrich manipulation tasks from tactile and vision through selfsupervision [42]. Another work used 6-axis force-torque and tactile sensors on a multi-finger rigid hand to train temporal neural-network models for in-hand manipulation tasks [43]. Opposed to contact sensing, a different approach used acoustic perception to gain information regarding hand-object contact [44].…”

Section: Related Workmentioning

confidence: 99%

Learning Haptic-based Object Pose Estimation for In-hand Manipulation with Underactuated Robotic Hands

Azulay¹,

Ben-David²,

Sintov³

2022

Preprint

View full text Add to dashboard Cite

Unlike traditional robotic hands, underactuated compliant hands are challenging to model due to inherent uncertainties. Consequently, pose estimation of a grasped object is usually performed based on visual perception. However, visual perception of the hand and object can be limited in occluded or partly-occluded environments. In this paper, we aim to explore the use of haptics, i.e., kinesthetic and tactile sensing, for pose estimation and in-hand manipulation with underactuated hands. Such haptic approach would mitigate occluded environments where line-of-sight is not always available. We put an emphasis on identifying the feature state representation of the system that does not include vision and can be obtained with simple and low-cost hardware. For tactile sensing, therefore, we propose a low-cost and flexible sensor that is mostly 3D printed along with the finger-tip and can provide implicit contact information. Taking a two-finger underactuated hand as a test-case, we analyze the contribution of kinesthetic and tactile features along with various regression models to the accuracy of the predictions. Furthermore, we propose a Model Predictive Control (MPC) approach which utilizes the pose estimation to manipulate objects to desired states solely based on haptics. We have conducted a series of experiments that validate the ability to estimate poses of various objects with different geometry, stiffness and texture, and show manipulation to goals in the workspace with relatively high accuracy.

show abstract

Variable In-Hand Manipulations for Tactile-Driven Robot Hand via CNN-LSTM

Cited by 10 publications

References 32 publications

Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Robotic Manipulation Planning for Automatic Peeling of Glass Substrate Based on Online Learning Model Predictive Path Integral

Learning Haptic-based Object Pose Estimation for In-hand Manipulation with Underactuated Robotic Hands

Contact Info

Product

Resources

About