A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation

Antonova, Rika; Yang, Jingyun; Sundaresan, Priya; Fox, Dieter; Ramos, Fábio; Bohg, Jeannette

doi:10.1109/lra.2022.3157377

Cited by 12 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deformable object manipulation is a long-standing challenge in robotics. However, many existing methods focus on objects such as ropes (Wang et al, 2019; Nair et al, 2017; Yan et al, 2020; Sundaresan et al, 2020; Lee et al, 2021; Antonova et al, 2022; Yan et al 2020, 2020), cables (She et al, 2021; Sanchez et al, 2020; Seita et al, 2021), clothes (Antonova et al, 2021a, 2021b; Ha and Song, 2022; Lin et al, 2021; Matas et al, 2018; Miller et al, 2012; Wu et al, 2020), and gauze (Thananjeyan et al, 2017). By contrast, we investigate the less-explored manipulation of elasto-plastic objects, such as plasticine, which are only studied in a limited number of works (Sanchez et al, 2018; Li et al, 2018; Lin et al).…”

Section: Related Workmentioning

confidence: 99%

RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks

Shi,

Xu,

Huang

et al. 2023

The International Journal of Robotics Research

View full text Add to dashboard Cite

Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degrees of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, for example, representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensory data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model predictive control (MPC) algorithms to plan the robot’s behavior. We show through experiments that with just 10 min of real-world robot interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various complex target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot’s manipulation capabilities.

show abstract

Section: Related Workmentioning

confidence: 99%

RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks

Shi,

Xu,

Huang

et al. 2023

The International Journal of Robotics Research

View full text Add to dashboard Cite

show abstract

“…Deformable object manipulation is a long-standing challenge in robotics. However, many existing methods focus on objects such as ropes [59,45,61,57,24,3,61], cables [54,48,53], clothes [31,38,36,60,15,2,1], and gauze [58]. By contrast, we investigate the less-explored manipulation of elasto-plastic objects, such as plasticine, which are only studied in limited previous works [49,27].…”

Section: B Manipulating Deformable Objectsmentioning

confidence: 99%

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Shi¹,

Xu²,

Huang³

et al. 2022

Robotics: Science and Systems XVIII

View full text Add to dashboard Cite

Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned modelbased planning framework is comparable to and sometimes better than human subjects on the tested tasks. 1

show abstract

“…Recent studies have attempted to address challenges in system identification and high computation costs in conventional continuum mechanics [6,7,8,9,10,11,12,13]. To ease the burden of system identification, prior studies infer the physics models' parameters based on high-fidelity physics engines [20,21,22] and simple force-deformation relationship (e.g. Hooke's law) [23].…”

Section: Related Workmentioning

confidence: 99%

“…3D geometric representation of deformable objects. Most recent studies have adopted discrete geometry representation for deformable objects with finite resolution such as meshes [7,10,28] and key points [15,21], sometimes accompanied with graph neural networks [29,27]. These approaches have been used largely for 1D and 2D objects (rope, cloth, top view of cylindrical objects).…”

Section: Related Workmentioning

confidence: 99%

VIRDO: Visio-tactile Implicit Representations of Deformable Objects

Wi¹,

Florence²,

Zeng³

et al. 2022

2022 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief distribution of deformation within a trajectory, and (ii) enables practical real-world application by removing the need for contact patches. In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.

show abstract

A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation

Cited by 12 publications

References 26 publications

RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks

RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networks

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

VIRDO: Visio-tactile Implicit Representations of Deformable Objects

Contact Info

Product

Resources

About