6IMPOSE: bridging the reality gap in 6D pose estimation for robotic grasping

Abstract

6D pose recognition has been a crucial factor in the success of robotic grasping, and recent deep learning based approaches have achieved remarkable results on benchmarks. However, their generalization capabilities in real-world applications remain unclear. To overcome this gap, we introduce 6IMPOSE, a novel framework for sim-to-real data generation and 6D pose estimation. 6IMPOSE consists of four modules: First, a data generation pipeline that employs the 3D software suite Blender to create synthetic RGBD image datasets with 6D pose annotations. Second, an annotated RGBD dataset of five household objects was generated using the proposed pipeline. Third, a real-time two-stage 6D pose estimation approach that integrates the object detector YOLO-V4 and a streamlined, real-time version of the 6D pose estimation algorithm PVN3D optimized for time-sensitive robotics applications. Fourth, a codebase designed to facilitate the integration of the vision system into a robotic grasping experiment. Our approach demonstrates the efficient generation of large amounts of photo-realistic RGBD images and the successful transfer of the trained inference model to robotic grasping experiments, achieving an overall success rate of 87% in grasping five different household objects from cluttered backgrounds under varying lighting conditions. This is made possible by fine-tuning data generation and domain randomization techniques and optimizing the inference pipeline, overcoming the generalization and performance shortcomings of the original PVN3D algorithm. Finally, we make the code, synthetic dataset, and all the pre-trained models available on GitHub.

A Method for Unseen Object Six Degrees of Freedom Pose Estimation Based on Segment Anything Model and Hybrid Distance Optimization

“…Evaluation Metrics. In this section, we employ two metrics, ADD (Average Distance of Model Points) and ADD-S (Average Distance of Model Points for Symmetric objects) [18,29,32,33] to assess the accuracy of 6-DoF pose estimation. The ADD metric quantifies the average Euclidean distance between corresponding points in the point cloud under the actual and predicted poses, calculating the mean discrepancy of each point's transformed location in the point cloud.…”

“…They focused on the transition from synthetic to real-world data, the so-called Sim2Real domain gap, achieving unsupervised domain adaptation through deformation feature matching with category shape priors. The 6IMPOSE framework [32], integrating a synthetic RGBD dataset generated by Blender with a target detection network based on YOLO-V4 and a lightweight pose estimation network, has propelled the advancement of real-time pose estimation. Despite these developments, 6-DoF object pose estimation based on RGB-D data still confronts numerous challenges.…”

Deep Learning-Based 6-DoF Object Pose Estimation Considering Synthetic Dataset

“…They also proposed sharing a separate self-coder for each object in a multi-object scenario [ 26 ]. Furthermore, the proposed 6IMPOSE [ 27 ] framework validates the efficiency of the position estimation model acquired from synthetic data training in real scenes. However, the accuracy of the object 6-DoF position estimation method, based on synthetic data, is notably lower than that based on real data on the datasets [ 34 , 35 ].…”

“…The proposed method demonstrated relatively excellent results on the T-LESS dataset. Furthermore, to solve the problem of poor generalization ability of synthetic data- based object 6-DoF pose estimation for practical applications, the 6IMPOSE [ 27 ] technique overcame the shortcomings of the PVN3D [ 21 ] algorithm in terms of generalization and performance, achieving better results in synthetic data-based pose estimation algorithms and confirming the validity of the synthetic data developed for real scenes.…”

“…Advances in DL techniques have led to significant progress not only in the areas of target detection [ 1 , 2 , 3 ] and image segmentation [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ], but also significant progress has been made in pose estimation using these techniques. They can be classified based on the types of datasets into (1) approaches relying on real datasets [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]; and (2) approaches based on synthetic data [ 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ]. However, the need for labeled real datasets raises a challenge due to the time-consuming and labor-intensive nature of their production, resulting in high dataset production costs [ 33 ].…”