Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics

Schwarz, Max; Behnke, Sven

doi:10.1109/icra40945.2020.9197309

Cited by 32 publications

(20 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2). Our approach is composed of three main components: a renderer [10], a deep CNN, and a shape space. The renderer is in charge of generating 2D images of realistic 3D models which will be used to train the CNN.…”

Section: Methodsmentioning

confidence: 99%

“…The use of synthetic data to generate training samples has been widely adopted in the deep learning community for object detection, semantic segmentation and pose estimation tasks [10][11][12][13]. One of the first successful attempts was proposed by Tobin et al [14], who trained an object detection network using only synthetic data and were able to transfer the network to real-world applications.…”

Section: Related Work a Rendering For Deep Learningmentioning

confidence: 99%

See 1 more Smart Citation

Category-Level 3D Non-Rigid Registration from Single-View RGB Images

Rodríguez¹,

Huber²,

Behnke³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a novel approach to solve the 3D non-rigid registration problem from RGB images using Convolutional Neural Networks (CNNs). Our objective is to find a deformation field (typically used for transferring knowledge between instances, e.g., grasping skills) that warps a given 3D canonical model into a novel instance observed by a single-view RGB image. This is done by training a CNN that infers a deformation field for the visible parts of the canonical model and by employing a learned shape (latent) space for inferring the deformations of the occluded parts. As result of the registration, the observed model is reconstructed. Because our method does not need depth information, it can register objects that are typically hard to perceive with RGB-D sensors, e.g. with transparent or shiny surfaces. Even without depth data, our approach outperforms the Coherent Point Drift (CPD) registration method for the evaluated object categories.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Work a Rendering For Deep Learningmentioning

confidence: 99%

Category-Level 3D Non-Rigid Registration from Single-View RGB Images

Rodríguez¹,

Huber²,

Behnke³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…While these methods show great results, it requires domain knowledge to find good parameters. With regards to creating photo-realistic images, there are also lots of research efforts [7], [16], [17], [18]. These sources prove that the qualities of the object model, such as textures/materials, and the renderers, including shadowing/lighting, are all important for sim2real.…”

Section: Related Work a Learning With Synthetic Datamentioning

confidence: 99%

Sim2Real Instance-Level Style Transfer for 6D Pose Estimation

Ikeda¹,

Tanishige²,

Amma³

et al. 2022

Preprint

View full text Add to dashboard Cite

In recent years, synthetic data has been widely used in the training of 6D pose estimation networks, in part because it automatically provides perfect annotation at low cost. However, there are still non-trivial domain gaps, such as differences in textures/materials, between synthetic and real data. These gaps have a measurable impact on performance.To solve this problem, we introduce a simulation to reality (sim2real) instance-level style transfer for 6D pose estimation network training. Our approach transfers the style of target objects individually, from synthetic to real, without human intervention. This improves the quality of synthetic data for training pose estimation networks. We also propose a complete pipeline from data collection to the training of a pose estimation network and conduct extensive evaluation on a real-world robotic platform. Our evaluation shows significant improvement achieved by our method in both pose estimation performance and the realism of images adapted by the style transfer.

show abstract

“…8) converts our parametrized world model into 3D point clouds that are suitable for point-to-point registration with the measurements P C s of the Velodyne 3D LiDAR, which is moved to capture a dense 3D scan of the pile or brick scene. We render the parametrized world model using an OpenGL-based renderer (Schwarz and Behnke, 2020) and obtain the point cloud P C m . Both point clouds are represented in the base-link B.…”

Section: Rendering and Samplingmentioning

confidence: 99%

Team NimbRo's UGV Solution for Autonomous Wall Building and Fire Fighting at MBZIRC 2020

Lenz¹,

Quenzel²,

Periyasamy³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Autonomous robotic systems for various applications including transport, mobile manipulation, and disaster response are becoming more and more complex. Evaluating and analyzing such systems is challenging. Robotic competitions are designed to benchmark complete robotic systems on complex state-of-the-art tasks. Participants compete in defined scenarios under equal conditions. We present our UGV solution developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Our hard-and software components to address the challenge tasks of wall building and fire fighting are integrated into a fully autonomous system. The robot consists of a wheeled omnidirectional base, a 6 DoF manipulator arm equipped with a magnetic gripper, a highly efficient storage system to transport box-shaped objects, and a water spraying system to fight fires. The robot perceives its environment using 3D LiDAR as well as RGB and thermal camera-based perception modules, is capable of picking box-shaped objects and constructing a pre-defined wall structure, as well as detecting and localizing heat sources in order to extinguish potential fires. A high-level planner solves the challenge tasks using the robot skills. We analyze and discuss our successful participation during the MBZIRC 2020 finals, present further experiments, and provide insights to our lessons learned.

show abstract

Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics

Cited by 32 publications

References 21 publications

Category-Level 3D Non-Rigid Registration from Single-View RGB Images

Category-Level 3D Non-Rigid Registration from Single-View RGB Images

Sim2Real Instance-Level Style Transfer for 6D Pose Estimation

Team NimbRo's UGV Solution for Autonomous Wall Building and Fire Fighting at MBZIRC 2020

Contact Info

Product

Resources

About