Object Shell Reconstruction: Camera-centric Object Representation for Robotic Grasping

Abstract

Robots can effectively grasp and manipulate objects using their 3D models. In this paper, we propose a simple shape representation and a reconstruction method that outperforms state-of-the-art methods in terms of geometric metrics and enables grasp generation with high precision and success. Our reconstruction method models the object geometry as a pair of depth images, composing the "shell" of the object. This representation allows using image-to-image residual ConvNet architectures for 3D reconstruction, generates object reconstruction directly in the camera frame, and generalizes well to novel object types. Moreover, an object shell can be converted into an object mesh in a fraction of a second, providing time and memory efficient alternative to voxel or implicit representations. We explore the application of shell representation for grasp planning. With rigorous experimental validation, both in simulation and on a real setup, we show that shell reconstruction encapsulates sufficient geometric information to generate precise grasps and the associated grasp quality with over 90% accuracy. Diverse grasps computed on shell reconstructions allow the robot to select and execute grasps in cluttered scenes with more than 93% success rate.

Digital Twin (DT)-CycleGAN: Enabling Zero-Shot Sim-to-Real Transfer of Visual Grasping Models

“…Recently, visionbased robotic grasping has attracted increasing attention as a central research theme due to the fast advancements of deep learning and computer vision, and has achieved remarkable progress [5], [19]. In general, visual grasping approaches can be categorized into two main streams: object reconstruction based methods and end-to-end methods [19], [20]. In general, 3D reconstruction of free-form objects can enable accurate grasp planning, while end-to-end methods can generate grasp proposals directly from the camera sensor [20].…”

“…In general, visual grasping approaches can be categorized into two main streams: object reconstruction based methods and end-to-end methods [19], [20]. In general, 3D reconstruction of free-form objects can enable accurate grasp planning, while end-to-end methods can generate grasp proposals directly from the camera sensor [20]. By sampling the grasp candidates or generating suitable grasps, end-toend methods demonstrated promising results in generalizing to even unseen and novel objects/backgrounds, and have gained increasing popularity [5].…”

Deep Learning Approaches to Grasp Synthesis: A Review

“…Kiatos et al [135] use a variational autoencoder [136] to predict the occluded surface points and associated normals of a partial 3D point cloud. Chavan-Dafle et al [137] predicted the depth image that estimates the 'back' side of an object from a masked depth image. The front and back sides can then be stitched together quickly to form an object mesh.…”