A system for generating complex physically accurate sensor images for automotive applications

Abstract

We describe an open-source simulator that creates sensor irradiance and sensor images of typical automotive scenes in urban settings. The purpose of the system is to support camera design and testing for automotive applications. The user can specify scene parameters (e.g., scene type, road type, traffic density, time of day) to assemble a large number of random scenes from graphics assets stored in a database. The sensor irradiance is generated using quantitative computer graphics methods, and the sensor images are created using image systems sensor simulation. The synthetic sensor images have pixel level annotations; hence, they can be used to train and evaluate neural networks for imaging tasks, such as object detection and classification. The end-to-end simulation system supports quantitative assessment -from scene to camera to network accuracy -for automotive applications.

Neural Network Generalization: The Impact of Camera Parameters

“…This approach synthesizes images as a form of domain adaptation. A third purpose is to co-design cameras and networks [3], [25], [26]. The goal of this application is to explore camera design that spans a wide range parameters, exceeding the consumer photography cameras, in order to discover systems whose images improve the accuracy of object detection networks.…”

“…Simulated scene radiance data and sensor irradiance were generated for a collection of 4000 city scenes, using the ISET3d software 4 [26]. This software defines the positions of vehicles and pedestrians that match traffic statistics using the Simulation of Urban MObility (SUMO) software package 5 [21].…”

“…Simulation of image data from the novel camera design is the only option. For this reason we are developing software tools (ISETAuto) [3], [25], [26]) for simulating physically accurate, complex and realistic driving scenes, modeling camera optics and sensors, and controlling the neural networks that detect and localize objects.…”

Soft Prototyping Camera Designs for Car Detection Based on a Convolutional Neural Network

“…The paper by [1] did not include procedural modeling and was restricted to relatively simple scenes; rendering was based on a version of PBRT [26] that was subsequently improved with regards to material modeling. The paper by [24] used Faster RCNN that was pre-trained using camera data from BDD100k [36] and tested on ISET3D synthetic data. The work described here uses (a) a much larger and more complex collection of synthetic scenes, (b) network training with the appropriate synthetic data, (c) a new network, Mask R-CNN with a ResNet backbone, and (d) more extensive camera algorithm analyses.…”

“…A prototyping system must combine quantitative computer graphics for creating accurate scene radiance with quantitative methods for simulating the imaging system. Such a system can simulate realistic camera images with accurate labels at each pixel ( Figure 1); these images can be used to train neural networks for object recognition and detection [33,24].…”