DefGraspNets: Grasp Planning on 3D Fields with Graph Neural Nets

Abstract

Robotic grasping of 3D deformable objects is critical for real-world applications such as food handling and robotic surgery. Unlike rigid and articulated objects, 3D deformable objects have infinite degrees of freedom. Fully defining their state requires 3D deformation and stress fields, which are exceptionally difficult to analytically compute or experimentally measure. Thus, evaluating grasp candidates for grasp planning typically requires accurate, but slow 3D finite element method (FEM) simulation. Sampling-based grasp planning is often impractical, as it requires evaluation of a large number of grasp candidates. Gradient-based grasp planning can be more efficient, but requires a differentiable model to synthesize optimal grasps from initial candidates. Differentiable FEM simulators may fill this role, but are typically no faster than standard FEM. In this work, we propose learning a predictive graph neural network (GNN), DefGraspNets, to act as our differentiable model. We train DefGraspNets to predict 3D stress and deformation fields based on FEM-based grasp simulations. DefGraspNets not only runs up to 1500x faster than the FEM simulator, but also enables fast gradient-based grasp optimization over 3D stress and deformation metrics. We design DefGraspNets to align with real-world grasp planning practices and demonstrate generalization across multiple test sets, including real-world experiments.

Graph Learning in Robotics: A Survey

“…Particles and keypoints are often inferred from visual representation, such as point clouds or mesh [50], images [31], [51] and depth images [48]. Recently, directly using mesh representations has gained attention as an alternative approach to model complex systems, especially for deformable objects [35], [36], [56], [57]. Among the others, it is worth discussing firstly MeshGraphNets [35], a general framework able to learn the dynamics of a wide range of systems, from cloths to fluids.…”

“…In a similar fashion, but in the context of manipulation planning, DefGraspNets [36] propose a multigraph to encode the object mesh together with the gripper mesh, which is then used to learn to predict the deformation of soft objects during manipulation.…”