Offline-Online Learning of Deformation Model for Cable Manipulation With Graph Neural Networks

Wang, Changhao; Zhang, Yuyou; Zhang, Xiang; Wang, Zheng; Zhu, Xiaojin; Jin, Shiyu; Tian, Tao; Tomizuka, Masayoshi

doi:10.1109/lra.2022.3158376

Cited by 36 publications

(18 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To achieve efficient manipulation policy learning, Lin et al [25] have used the positions of keypoints on the rope as the reduced states. To bridge the sim-to-real gap, Wang et al [20] treated keypoints as nodes in a graph and designed an offline-online learning framework based on graph neural networks. Ma et al [21] designed a graph neural network to learn the forward dynamic model of the deformable objects and achieved precise visual manipulation.…”

Section: Manipulation Policy Learning From Keypointsmentioning

confidence: 99%

Learning visual-based deformable object rearrangement with local graph neural networks

Deng

Wang

Chen

2023

Complex Intell. Syst.

View full text Add to dashboard Cite

Goal-conditioned rearrangement of deformable objects (e.g. straightening a rope and folding a cloth) is one of the most common deformable manipulation tasks, where the robot needs to rearrange a deformable object into a prescribed goal configuration with only visual observations. These tasks are typically confronted with two main challenges: the high dimensionality of deformable configuration space and the underlying complexity, nonlinearity and uncertainty inherent in deformable dynamics. To address these challenges, we propose a novel representation strategy that can efficiently model the deformable object states with a set of keypoints and their interactions. We further propose local-graph neural network (GNN), a light local GNN learning to jointly model the deformable rearrangement dynamics and infer the optimal manipulation actions (e.g. pick and place) by constructing and updating two dynamic graphs. Both simulated and real experiments have been conducted to demonstrate that the proposed dynamic graph representation shows superior expressiveness in modeling deformable rearrangement dynamics. Our method reaches much higher success rates on a variety of deformable rearrangement tasks (96.3% on average) than state-of-the-art method in simulation experiments. Besides, our method is much more lighter and has a 60% shorter inference time than state-of-the-art methods. We also demonstrate that our method performs well in the multi-task learning scenario and can be transferred to real-world applications with an average success rate of 95% by solely fine tuning a keypoint detector. A supplementary video can be found at https://youtu.be/AhwTQo6fCM0.

show abstract

Section: Manipulation Policy Learning From Keypointsmentioning

confidence: 99%

Learning visual-based deformable object rearrangement with local graph neural networks

Deng

Wang

Chen

2023

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…Recently, increasing attention has been put to the deformation learning approaches that embed geometric graph structures into neural architectures. Methods have been proposed using mesh-based (Pfaff et al (2021)) and particle-based (Shi et al (2022); Wang et al (2022)) GNNs to learn deformation dynamics. The learned deformation model can be used to predict deformation for planning manipulation actions.…”

Section: Related Workmentioning

confidence: 99%

Modal-graph 3D shape servoing of deformable objects with raw point clouds

Yang,

Sui,

Zhong

et al. 2023

The International Journal of Robotics Research

View full text Add to dashboard Cite

Deformable object manipulation (DOM) with point clouds has great potential as nonrigid 3D shapes can be measured without detecting and tracking image features. However, robotic shape control of deformable objects with point clouds is challenging due to: the unknown point correspondences and the noisy partial observability of raw point clouds; the modeling difficulties of the relationship between point clouds and robot motions. To tackle these challenges, this paper introduces a novel modal-graph framework for the model-free shape servoing of deformable objects with raw point clouds. Unlike the existing works studying the object’s geometry structure, we propose a modal graph to describe the low-frequency deformation structure of the DOM system, which is robust to the measurement irregularities. The modal graph enables us to directly extract low-dimensional deformation features from raw point clouds without extra processing of registrations, refinements, and occlusion removal. It also preserves the spatial structure of the DOM system to inverse the feature changes into robot motions. Moreover, as the framework is built with unknown physical and geometric object models, we design an adaptive robust controller to deform the object toward the desired shape while tackling the modeling uncertainties, noises, and disturbances online. The system is proved to be input-to-state stable (ISS) using Lyapunov-based methods. Extensive experiments are conducted to validate our method using linear, planar, tubular, and volumetric objects under different settings.

show abstract

“…Compared to tracking experiments in simulation, noisy points in point cloud segmentation and the sim-to-real gap in the trained autoencoder result in larger errors. However, point cloud segmentation is not the main focus of the paper and we can narrow the sim-to-real gap by collecting real-world datasets to fine tune the autoencoder [8].…”

Section: Tracking Dlos In Real Worldmentioning

confidence: 99%

“…One common manipulation task is to control the shape of a DLO to achieve a certain goal state, such as wires routing [3]- [5] and kinematic control for soft robots [6]. In these applications, reliable DLO state tracking is important for feedback in closed-loop classical control and data collection in training data-driven model-based control [7], [8]. However, tracking DLOs is still an open research question compared to the achievements in rigid object tracking [2].…”

Section: Introductionmentioning

confidence: 99%

Particle Filters in Latent Space for Robust Deformable Linear Object Tracking

Yang

Stork

Stoyanov

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Tracking of deformable linear objects (DLOs) is important for many robotic applications. However, achieving robust and accurate tracking is challenging due to the lack of distinctive features or appearance on the DLO, the object's high-dimensional state space, and the presence of occlusion. In this letter, we propose a method for tracking the state of a DLO by applying a particle filter approach within a lower-dimensional state embedding learned by an autoencoder. The dimensionality reduction preserves state variation, while simultaneously enabling a particle filter to accurately track DLO state evolution with a practically feasible number of particles. Compared to previous works, our method requires neither running a high-fidelity physics simulation, nor manual designs of constraints and regularization. Without the assumption of knowing the initial DLO state, our method can achieve accurate tracking even under complex DLO motions and in the presence of severe occlusions. Our implementation is available at https://amm.aass.oru.se/dlo-pf-tracking/.

show abstract

Offline-Online Learning of Deformation Model for Cable Manipulation With Graph Neural Networks

Cited by 36 publications

References 28 publications

Learning visual-based deformable object rearrangement with local graph neural networks

Learning visual-based deformable object rearrangement with local graph neural networks

Modal-graph 3D shape servoing of deformable objects with raw point clouds

Particle Filters in Latent Space for Robust Deformable Linear Object Tracking

Contact Info

Product

Resources

About