Object Detection Using Sim2Real Domain Randomization for Robotic Applications

Abstract

Robots working in unstructured environments must be capable of sensing and interpreting their surroundings. One of the main obstacles of deep-learning-based models in the field of robotics is the lack of domain-specific labeled data for different industrial applications. In this article, we propose a sim2real transfer learning method based on domain randomization for object detection with which labeled synthetic datasets of arbitrary size and object types can be automatically generated. Subsequently, a state-of-the-art convolutional neural network, YOLOv4, is trained to detect the different types of industrial objects. With the proposed domain randomization method, we could shrink the reality gap to a satisfactory level, achieving 86.32% and 97.38% mAP 50 scores, respectively, in the case of zero-shot and one-shot transfers, on our manually annotated dataset containing 190 real images. Our solution fits for industrial use as the data generation process takes less than 0.5 s per image and the training lasts only around 12 h, on a GeForce RTX 2080 Ti GPU. Furthermore, it can reliably differentiate similar classes of objects by having access to only one real image for training. To our best knowledge, this is the only work thus far satisfying these constraints.

Towards a process for the creation of synthetic training data for AI-computer vision models utilizing engineering data

“…In this approach, a model trained predominantly on synthetic data is enhanced by incorporating a subset of real-world data during its training phase. Multiple studies have demonstrated that adding a relatively small set of real training images into a largely synthetic data set can significantly improve the models ability to generalize in the target domain (Horváth et al, 2023;Tremblay et al, 2018). However, using realistic renderings to handle the domain gap is also investigated in literature (Denninger et al, 2019).…”

Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer

“…and hope that the randomization captures the distribution of the real-world data. This technique has been applied to object detection and grasping [2,18,21,47,49], and pose estimation [26,28,29,34,36,46,50,56]. The randomization is usually tuned empirically hence it is not efficient.…”

Towards Object Agnostic and Robust 4-DoF Table-Top Grasping

“…However, the availability of labeled training data remains a major challenge as the process of data labeling is labor-intensive and costlier [9]. To overcome this, many solutions have used simulations for auto-generating the training data [4], [10] followed by sim-to-real transfer of the learning for real-world deployment [11]. Previously, It was believed that real-world visuals differ significantly from the simulated world and hence the sim-to-real transfer is not promising in the case of models trained with only synthetic RGB data [12], [13].…”