Style-transfer GANs for bridging the domain gap in synthetic pose estimator training

Rojtberg, Pavel; Pöllabauer, Thomas; Kuijper, Arjan

doi:10.1109/aivr50618.2020.00039

Cited by 15 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They specifically focused on applying domain randomization to the renderings and creating images with good viewpoint coverage. Rojtberg et al [ 40 ] utilized GANs to learn the difference between real and synthetic images and then transform synthetic images into the real domain based on these networks. They found that this strategy cannot reach the performance of real images, but increases performance compared to pure domain randomization.…”

Section: Related Workmentioning

confidence: 99%

A Survey of 6D Object Detection Based on 3D Models for Industrial Applications

2022

Self Cite

View full text Add to dashboard Cite

Six-dimensional object detection of rigid objects is a problem especially relevant for quality control and robotic manipulation in industrial contexts. This work is a survey of the state of the art of 6D object detection with these use cases in mind, specifically focusing on algorithms trained only with 3D models or renderings thereof. Our first contribution is a listing of requirements typically encountered in industrial applications. The second contribution is a collection of quantitative evaluation results for several different 6D object detection methods trained with synthetic data and the comparison and analysis thereof. We identify the top methods for individual requirements that industrial applications have for object detectors, but find that a lack of comparable data prevents large-scale comparison over multiple aspects.

show abstract

Section: Related Workmentioning

confidence: 99%

A Survey of 6D Object Detection Based on 3D Models for Industrial Applications

2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…CycleGAN [9] can translate unpaired images bidirectionally between two domains. For example, Rojtberg et al [8] leverages the CycleGAN [9] to get a mapping between synthetic and real domain for pose estimation. Moreover, DRIT [12] and MUNIT [11] can handle multiple latent spaces for image-to-image translation.…”

Section: B Domain Adaptationmentioning

confidence: 99%

“…One popular approach here is the use of Generative Adversarial Networks (GANs). For example, Rojtberg et al [8] leveraged Cycle-GAN [9] to transfer synthetic images to realistic ones for 6D pose estimation networks. However, as described in INIT [10], if the target domain is a complex scene containing multiple objects, serious inconsistencies will occur because these methods [9], [11], [12] focus on directly adapting a global style to the entire image.…”

Section: Introductionmentioning

confidence: 99%

Sim2Real Instance-Level Style Transfer for 6D Pose Estimation

Ikeda¹,

Tanishige²,

Amma³

et al. 2022

Preprint

View full text Add to dashboard Cite

In recent years, synthetic data has been widely used in the training of 6D pose estimation networks, in part because it automatically provides perfect annotation at low cost. However, there are still non-trivial domain gaps, such as differences in textures/materials, between synthetic and real data. These gaps have a measurable impact on performance.To solve this problem, we introduce a simulation to reality (sim2real) instance-level style transfer for 6D pose estimation network training. Our approach transfers the style of target objects individually, from synthetic to real, without human intervention. This improves the quality of synthetic data for training pose estimation networks. We also propose a complete pipeline from data collection to the training of a pose estimation network and conduct extensive evaluation on a real-world robotic platform. Our evaluation shows significant improvement achieved by our method in both pose estimation performance and the realism of images adapted by the style transfer.

show abstract

“…In addition to the aforementioned approaches, domain adaptation techniques can be used to further bridge the domain gap between synthetic and real images. Generative adversarial networks (GANs) [ 36 ] can be used to transform generated synthetic images closer to the target domain [ 37 , 38 , 39 , 40 ]. Alternatively, both source and target domain can be transformed into an intermediate domain, e.g., with the Laplacian Filter [ 40 ] or the Pencil Filter [ 41 , 42 ].…”

Section: Related Workmentioning

confidence: 99%

“…Generative adversarial networks (GANs) [ 36 ] can be used to transform generated synthetic images closer to the target domain [ 37 , 38 , 39 , 40 ]. Alternatively, both source and target domain can be transformed into an intermediate domain, e.g., with the Laplacian Filter [ 40 ] or the Pencil Filter [ 41 , 42 ].…”

Section: Related Workmentioning

confidence: 99%

Generating Images with Physics-Based Rendering for an Industrial Object Detection Task: Realism versus Domain Randomization

Eversberg

Lambrecht

2021

Sensors

View full text Add to dashboard Cite

Limited training data is one of the biggest challenges in the industrial application of deep learning. Generating synthetic training images is a promising solution in computer vision; however, minimizing the domain gap between synthetic and real-world images remains a problem. Therefore, based on a real-world application, we explored the generation of images with physics-based rendering for an industrial object detection task. Setting up the render engine’s environment requires a lot of choices and parameters. One fundamental question is whether to apply the concept of domain randomization or use domain knowledge to try and achieve photorealism. To answer this question, we compared different strategies for setting up lighting, background, object texture, additional foreground objects and bounding box computation in a data-centric approach. We compared the resulting average precision from generated images with different levels of realism and variability. In conclusion, we found that domain randomization is a viable strategy for the detection of industrial objects. However, domain knowledge can be used for object-related aspects to improve detection performance. Based on our results, we provide guidelines and an open-source tool for the generation of synthetic images for new industrial applications.

show abstract

Style-transfer GANs for bridging the domain gap in synthetic pose estimator training

Cited by 15 publications

References 16 publications

A Survey of 6D Object Detection Based on 3D Models for Industrial Applications

A Survey of 6D Object Detection Based on 3D Models for Industrial Applications

Sim2Real Instance-Level Style Transfer for 6D Pose Estimation

Generating Images with Physics-Based Rendering for an Industrial Object Detection Task: Realism versus Domain Randomization

Contact Info

Product

Resources

About