TacGNN: Learning Tactile-Based In-Hand Manipulation With a Blind Robot Using Hierarchical Graph Neural Network

Abstract

In this paper, we propose a novel framework for tactile-based dexterous manipulation learning with a blind anthropomorphic robotic hand, i.e. without visual sensing. First, object-related states were extracted from the raw tactile signals by a graph-based perception model -TacGNN. The resulting tactile features were then utilized in the policy learning of an in-hand manipulation task in the second stage. This method was examined by a Baoding ball task -simultaneously manipulating two spheres around each other by 180 degrees in hand. We conducted experiments on object states prediction and in-hand manipulation using a reinforcement learning algorithm (PPO). Results show that TacGNN is effective in predicting objectrelated states during manipulation by decreasing the RMSE of prediction to 0.096cm comparing to other methods, such as MLP, CNN, and GCN. Finally, the robot hand could finish an in-hand manipulation task solely relying on the robotic own perception -tactile sensing and proprioception. In addition, our methods are tested on three tasks with different difficulty levels and transferred to the real robot without further training. https://sites.google.com/view/tacgnn

Globally Guided Deep V-Network-Based Motion Planning Algorithm for Fixed-Wing Unmanned Aerial Vehicles

“…[22] proposed an improved RRT* algorithm to realize 3D path planning. (2) Deep neural network-based learning methods in the field of robotics: In recent years, with the increase in computational power, deep neural network-based learning methods have shown great potential in dealing with high-dimensional and complex environmental states [23,24], especially in the field of robotics, such as robotic arms [25], wheeled and multi-legged robots [26][27][28], etc., which have been applied to some experimental real-world scenarios. For example, in [29], the authors proposed an imitation learning approach based on perceptual information to realize motion planning for UAVs, and Ref.…”

Bridging Locomotion and Manipulation Using Reconfigurable Robotic Limbs via Reinforcement Learning

“…While multilayer perceptron (MLP) models are widely adopted in building reinforcement learning models for efficient feature extraction from the data, recent development in graph neural networks (GNN) offers a different approach that considers the reactive and sensory architecture of the robotic topology for learning skills such as locomotion and manipulation [ 23 , 24 ]. For example, GNN-based RL policies explicitly parameterize the interaction between entities (i.e., joints or links) with neural networks.…”