DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition

Planche, Benjamin; Wu, Ziyan; Ma, Kai; Sun, Shanhui; Kluckner, Stefan; Lehmann, Oliver; Chen, Terrence; Hutter, Andreas; Zakharov, Sergey; Kosch, Harald; Ernst, Jan

doi:10.1109/3dv.2017.00011

Cited by 54 publications

(33 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically in [13], a 6D grasp pose detector has been trained in a simulator and transferred successfully to the real robot with 93% success rate without any special mechanisms used for transfer. In [24] it was shown that by enhancing simulated data to mimic the signal processing pipeline of a depth camera transfer can be further facilitated. We follow some of these ideas in this work.…”

Section: Related Workmentioning

confidence: 99%

Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images

Litvak

Biess

Bar-Hillel

2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Most of industrial robotic assembly tasks today require fixed initial conditions for successful assembly. These constraints induce high production costs and low adaptability to new tasks. In this work we aim towards flexible and adaptable robotic assembly by using 3D CAD models for all parts to be assembled. We focus on a generic assembly task -the Siemens Innovation Challenge -in which a robot needs to assemble a gearlike mechanism with high precision into an operating system. To obtain the millimeter-accuracy required for this task and industrial settings alike, we use a depth camera mounted near the robot's end-effector. We present a high-accuracy two-stage pose estimation procedure based on deep convolutional neural networks, which includes detection, pose estimation, refinement, and handling of near-and full symmetries of parts. The networks are trained on simulated depth images with means to ensure successful transfer to the real robot. We obtain an average pose estimation error of 2.16 millimeters and 0.64 degree leading to 91% success rate for robotic assembly of randomly distributed parts. To the best of our knowledge, this is the first time that the Siemens Innovation Challenge is fully addressed, with all the parts assembled with high success rates. Figure 1: The KUKA LBR IIWA robot performs the Siemens Innovation Challenge successfully in simulation (top row) and in reality (bottom row).

show abstract

Section: Related Workmentioning

confidence: 99%

Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images

Litvak

Biess

Bar-Hillel

2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

show abstract

“…Planche et al in [24] generate synthetic depth data from 3D models. In this work, they try to understand the causes of the noise in the real depth data and mimic this noise in synthetic depth data.…”

Section: Related Workmentioning

confidence: 99%

Sim-to-Real Learning for Casualty Detection from Ground Projected Point Cloud Data

Saputra

Rakicevic

Kormushev

2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

This paper addresses the problem of human body detection-particularly a human body lying on the ground (a.k.a. casualty)-using point cloud data. This ability to detect a casualty is one of the most important features of mobile rescue robots, in order for them to be able to operate autonomously. We propose a deep-learning-based casualty detection method using a deep convolutional neural network (CNN). This network is trained to be able to detect a casualty using a point-cloud data input. In the method we propose, the point cloud input is pre-processed to generate a depth image-like ground-projected heightmap. This heightmap is generated based on the projected distance of each point onto the detected ground plane within the point cloud data. The generated heightmap-in image form-is then used as an input for the CNN to detect a human body lying on the ground. To train the neural network, we propose a novel sim-to-real approach, in which the network model is trained using synthetic data obtained in simulation and then tested on real sensor data. To make the model transferable to real data implementations, during the training we adopt specific data augmentation strategies with the synthetic training data. The experimental results show that data augmentation introduced during the training process is essential for improving the performance of the trained model on real data. More specifically, the results demonstrate that the data augmentations on raw point-cloud data have contributed to a considerable improvement of the trained model performance.

show abstract

“…Though synthetic data has been extensively used to train deep CNN models for real-world images [49,18,52,37], this approach has been relatively limited for medical imaging.…”

Section: Synthetic Endoscopy Data For Trainingmentioning

confidence: 99%

Deep learning with cinematic rendering: fine-tuning deep neural networks using photorealistic medical images

et al. 2018

View full text Add to dashboard Cite

Deep learning has emerged as a powerful artificial intelligence tool to interpret medical images for a growing variety of applications. However, the paucity of medical imaging data with high-quality annotations that is necessary for training such methods ultimately limits their performance. Medical data is challenging to acquire due to privacy issues, shortage of experts available for annotation, limited representation of rare conditions and cost. This problem has previously been addressed by using synthetically generated data. However, networks trained on synthetic data often fail to generalize to real data. Cinematic rendering simulates the propagation and interaction of light passing through tissue models reconstructed from CT data, enabling the generation of photorealistic images. In this paper, we present one of the first applications of cinematic rendering in deep learning, in which we propose to fine-tune synthetic data-driven networks using cinematically rendered CT data for the task of monocular depth estimation in endoscopy. Our experiments demonstrate that: (a) convolutional neural networks (CNNs) trained on synthetic data and fine-tuned on photorealistic cinematically rendered data adapt better to real medical images and demonstrate more robust performance when compared to networks with no fine-tuning, (b) these fine-tuned networks require less training data to converge to an optimal solution, and (c) fine-tuning with data from a variety of photorealistic rendering conditions of the same scene prevents the network from learning patient-specific information and aids in generalizability of the model. Our empirical evaluation demonstrates that networks fine-tuned with cinematically rendered data predict depth with 56.87% less error for rendered endoscopy images and 27.49% less error for real porcine colon endoscopy images.

show abstract

DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition

Cited by 54 publications

References 49 publications

Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images

Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images

Sim-to-Real Learning for Casualty Detection from Ground Projected Point Cloud Data

Deep learning with cinematic rendering: fine-tuning deep neural networks using photorealistic medical images

Contact Info

Product

Resources

About