Learning Deformable Object Manipulation From Expert Demonstrations

Salhotra, Gautam; Liu, I-Chun Arthur; Dominguez-Kuhne, Marcus; Sukhatme, Gaurav S.

doi:10.1109/lra.2022.3187843

Cited by 18 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these model-based approaches require time-consuming planning like Cross Entropy Method or Model Predictive Control to achieve the goal cloth state. As for model-free approaches, some approaches learn task-specific cloth manipulation policies from expert demonstrations [7], [8], [9]. However, these task-specific policies fail to reuse information for different tasks efficiently.…”

Section: A Sequential Multi-step Cloth Manipulationmentioning

confidence: 99%

See 1 more Smart Citation

Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

Mo,

Xia,

Wang

et al. 2023

Preprint

View full text Add to dashboard Cite

Sequential multi-step cloth manipulation is a challenging problem in robotic manipulation, requiring a robot to perceive the cloth state and plan a sequence of chained actions leading to the desired state. Most previous works address this problem in a goal-conditioned way, and goal observation must be given for each specific task and cloth configuration, which is not practical and efficient. Thus, we present a novel multistep cloth manipulation planning framework named Foldformer. Foldformer can complete similar tasks with only a general demonstration and utilize a space-time attention mechanism to capture the instruction information behind this demonstration. We experimentally evaluate Foldsformer on four representative sequential multi-step manipulation tasks and show that Foldsformer significantly outperforms state-of-the-art approaches in simulation. Foldformer can complete multi-step cloth manipulation tasks even when configurations of the cloth (e.g., size and pose) vary from configurations in the general demonstrations. Furthermore, our approach can be transferred from simulation to the real world without additional training or domain randomization. Despite training on rectangular clothes, we also show that our approach can generalize to unseen cloth shapes (T-shirts and shorts). Videos and source code are available at: https://sites.google.com/view/foldsformer.

show abstract

Section: A Sequential Multi-step Cloth Manipulationmentioning

confidence: 99%

“…Some of these approaches learn task-specific sequential multi-step cloth manipulation policies from expert demonstrations. Although these approaches are demonstrated to be effective, different models should be trained to perform different tasks [7], [8], [9]. Others learn a goal-conditioned policy from random data and can achieve arbitrary cloth goal states.…”

mentioning

confidence: 99%

Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

Mo,

Xia,

Wang

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Prior works [40,41,42,43,44] propose methods to detect or model physics properties of granular media or liquids and test on scooping and pouring. In contrast, we propose a general method of predicting 3D tool flow which does not require modeling properties of deformables and is not specialized to scooping or pouring tasks, and which uses point clouds as inputs instead of RGB images [45].…”

Section: Related Workmentioning

confidence: 99%

“…Overall, these results may provide evidence for the benefits of imitation learning in these environments over reinforcement learning. An interesting future direction to explore for these tasks would be to combine imitation learning with reinforcement learning [45,69,70].…”

Section: Figures S2 and S3mentioning

confidence: 99%

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Seita¹,

Wang²,

Shetty³

et al. 2022

Preprint

View full text Add to dashboard Cite

Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense perpoint flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform 50 physical scooping experiments with ToolFlowNet and attain 82% scooping success. See https://tinyurl.com/toolflownet for supplementary material.

show abstract

“…In contrast, others attempt to reconstruct the corresponding mesh to guide the manipulation [10,155,233]. Policy noise [130], demonstration data [244,311], specially engineered data [156,386], HER [9], MaxEnt-RL objective [392,146,234] and advantage-weighted loss exploration term [311] are leverage to overcome the exploration problem for DRL applications.…”

Section: Data-driven Control In Cloth-shapingmentioning

confidence: 99%

Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: Present, Challenges, and Future Prospects

Kadi¹,

Terzic²

2022

Preprint

View full text Add to dashboard Cite

Manipulating cloth-like deformable objects (CDOs) is a long-standing problem in the robotics community. CDOs are flexible (non-rigid) objects that do not show a detectable level of com- pression strength while two points on the article are pushed towards each otherand include objects such as ropes (1D), fabrics (2D) and bags (3D). In general, CDOs’ many degrees of freedom (DoF) introduce severe self-occlusion and complex state-action dynamics as significant obstacles for perception and manipulation systems. These challenges exacerbate existing issues of modern robotic control methods such as imitation learning (IL) and reinforcement learning (RL). This review focuses on the application details of data-driven control methods on four major task families in this domain: cloth-shaping, rope manipulation, dressing and bag manip- ulation. Furthermore, we identify specific inductive biases in these four domains that present challenges for more general IL and RL algorithms, and summarise the future direction for the development of the field.

show abstract

Learning Deformable Object Manipulation From Expert Demonstrations

Cited by 18 publications

References 19 publications

Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: Present, Challenges, and Future Prospects

Contact Info

Product

Resources

About