Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

Zhang, Xiaoshuai; Chen, Rui; Xiang, Fanbo; Qin, Yuzhe; Gu, Jiwei; Li, Zhan; Liu, Minghua; Zeng, Peiyu; Han, Song-Fang; Huang, Zhiao; Mu, Tongzhou; Xu, Jing; Sheng, Hao

doi:10.48550/arxiv.2201.11924

Cited by 2 publications

(2 citation statements)

References 69 publications

(84 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work exists that seeks to optimize the individual models used in the virtual environment such that their renderings are as close as possible to the real object [27]. However, we argue that this is intractable on a large scale.…”

Section: B Scene Modelingmentioning

confidence: 94%

Camera simulation for robot simulation: how important are various camera model components?

Elmquist¹,

Serban²,

Negrut³

2022

Preprint

View full text Add to dashboard Cite

Modeling cameras for the simulation of autonomous robotics is critical for generating synthetic images with appropriate realism to effectively evaluate a perception algorithm in simulation. In many cases though, simulated images are produced by traditional rendering techniques that exclude or superficially handle processing steps and aspects encountered in the actual camera pipeline. The purpose of this contribution is to quantify the degree to which the exclusion from the camera model of various image generation steps or aspects affect the sim-to-real gap in robotics. We investigate what happens if one ignores aspects tied to processes from within the physical camera, e.g., lens distortion, noise, and signal processing; scene effects, e.g., lighting and reflection; and rendering quality. The results of the study demonstrate, quantitatively, that large-scale changes to color, scene, and location have far greater impact than model aspects concerned with local, feature-level artifacts. Moreover, we show that these scene-level aspects can stem from lens distortion and signal processing, particularly when considering white-balance and auto-exposure modeling.

show abstract

Section: B Scene Modelingmentioning

confidence: 94%

Camera simulation for robot simulation: how important are various camera model components?

Elmquist¹,

Serban²,

Negrut³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[25] presented a new differentiable structure-light depth sensor simulation pipeline, but cannot simulate the transparent material, limited by the renderer. Recently, [42] proposed a physics-grounded active stereovision depth sensor simulator for various sim-to-real applications, but focused on instance-level objects and the robot arm workspace. Our DREDS pipeline generates realistic RGBD images for various materials and scene environments, which can generalize the proposed model to category-level unseen object instances and novel categories.…”

Section: Depth Sensor Simulationmentioning

confidence: 99%

Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects

Dai¹,

Jiyao²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Commercial depth sensors usually generate noisy and missing depths, especially on specular and transparent objects, which poses critical issues to downstream depth or point cloud-based tasks. To mitigate this problem, we propose a powerful RGBD fusion network, Swin-DRNet, for depth restoration. We further propose Domain Randomization-Enhanced Depth Simulation (DREDS) approach to simulate an active stereo depth system using physically based rendering and generate a large-scale synthetic dataset that contains 130K photorealistic RGB images along with their simulated depths carrying realistic sensor noises. To evaluate depth restoration methods, we also curate a real-world dataset, namely STD, that captures 30 cluttered scenes composed of 50 objects with different materials from specular, transparent, to diffuse. Experiments demonstrate that the proposed DREDS dataset bridges the sim-to-real domain gap such that, trained on DREDS, our SwinDRNet can seamlessly generalize to other real depth datasets, e.g. ClearGrasp, and outperform the competing methods on depth restoration with a real-time speed. We further show that our depth restoration effectively boosts the performance of downstream tasks, including category-level pose estimation and grasping tasks. Our data and code are available at https://github.com/PKU-EPIC/DREDS.

show abstract

Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation

Cited by 2 publications

References 69 publications

Camera simulation for robot simulation: how important are various camera model components?

Camera simulation for robot simulation: how important are various camera model components?

Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects

Contact Info

Product

Resources

About