Photo-realistic Neural Domain Randomization

Abstract

Synthetic data is a scalable alternative to manual supervision, but it requires overcoming the sim-to-real domain gap. This discrepancy between virtual and real worlds is addressed by two seemingly opposed approaches: improving the realism of simulation or foregoing realism entirely via domain randomization. In this paper, we show that the recent progress in neural rendering enables a new unified approach we call Photo-realistic Neural Domain Randomization (PNDR). We propose to learn a composition of neural networks that acts as a physics-based ray tracer generating high-quality renderings from scene geometry alone. Our approach is modular, composed of different neural networks for materials, lighting, and rendering, thus enabling randomization of different key image generation components in a differentiable pipeline. Once trained, our method can be combined with other methods and used to generate photo-realistic image augmentations online and significantly more efficiently than via traditional ray-tracing. We demonstrate the usefulness of PNDR through two downstream tasks: 6D object detection and monocular depth estimation. Our experiments show that training with PNDR enables generalization to novel scenes and significantly outperforms the state of the art in terms of real-world transfer.

Hybrid synthetic data generation pipeline that outperforms real data

“…We have successfully tackled the very challenging ObjectNet dataset by training on purely synthetic data and managed to outperform real data on it, a feat that only two other works have achieved to date 38 , 56 . We have also beaten the state-of-the-art CNN classification performance for ObjectNet, with our 72% top-1 accuracy.…”

“…9, where the synthetic images are composed by combining 3D models with random textures, against a background of random images taken from the Flikr 8k. 37 A 2022 approach called photorealistic neural domain randomization (PNDR) 38 utilizes a neural rendering technique, which learns a combination of modular neural networks to generate high-quality renderings, randomizing different aspects of a scene including lighting and materials while still preserving realism.…”

“…A 2022 approach called photorealistic neural domain randomization (PNDR) 38 utilizes a neural rendering technique, which learns a combination of modular neural networks to generate high-quality renderings, randomizing different aspects of a scene including lighting and materials while still preserving realism.…”

Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos

“…Synthetic datasets are large scale (up to 818k objects of [13]), but require extra efforts to bridge the domain gap for applications on real imagery [52,39,48,19,38,35].…”

A Design and Implement of an Automatic Intelligent Car

“…PiFu [36] learns the occupancy field in 3D directly by the occupancy implicit functions. Some works learn shape codes for different types of object priors like CAD model [37], [38], SDF-based Database [39], activation code library [40]. Recently, single-view to 3D NeRF has emerged as a promising approach to tackling this problem, with impressive results achieved by methods such as MVSNerf [41] and PixelNerf [29] using multi-view supervision, obtaining multi-view images is often more expensive and impractical compared to single-view images, especially in real-world scenarios.…”