Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Rosenzweig, Julia; Brito, Eduardo; Kobialka, Hans-Ulrich; Akila, Maram; Schmidt, Nico M.; Schlicht, Peter; Schneider, Ján; Hüger, Fabian; Rottmann, Matthias; Houben, Sebastian; Wirtz, Tim

doi:10.1109/ivworkshops54471.2021.9669248

Cited by 4 publications

(2 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several works, see [10,11,13,15,16,22], have proposed using simulators for training and testing of DNNs. Among these works, [16] propose a label-to-image transfer method to remove the domain gap problem by generating paired datasets to test the performance of networks trained on real-world data. Paranjape et al [11] propose a simulation environment to generate scenarios for testing autonomous vehicles.…”

Section: Related Workmentioning

confidence: 99%

Semantic Concept Testing in Autonomous Driving by Extraction of Object-Level Annotations from CARLA

Gannamaneni

Houben

Akila

2021

2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

Self Cite

View full text Add to dashboard Cite

With the growing use of Deep Neural Networks (DNNs) in various safety-critical applications comes an increasing need for Verification and Validation (V&V) of these DNNs. Unlike testing in software engineering, where several established methods exist for V&V, DNN testing is still at an early stage. The data-driven nature of DNNs adds to the complexity of testing them. In the scope of autonomous driving, we showcase our validation method by leveraging objectlevel annotations (object metadata) to test DNNs on a more granular level using human-understandable semantic concepts like gender, shirt colour, age, and illumination. Such an enhanced granularity, as we detail, can prove useful in the construction of closed-loop testing or the investigation of dataset coverage/completeness. Our add-on sensor to the CARLA simulator enables us to generate datasets with this granular metadata. For the task of semantic segmentation for pedestrian detection using DeepLabv3+, we highlight potential insights and challenges that become apparent on this level of granularity. For instance, imbalances within a CARLA generated dataset w.r.t. the pedestrian distribution do not directly carry over into weak spots of the DNN performances and vice versa.

show abstract

Section: Related Workmentioning

confidence: 99%

Semantic Concept Testing in Autonomous Driving by Extraction of Object-Level Annotations from CARLA

Gannamaneni

Houben

Akila

2021

2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Procedural methods for road generation [PJX+20] can enhance the capabilities of these methods. Some methods try to reduce the remaining domain gap by synthetization of test images through generative approaches [RBK+21]. But similar to label noise the image inconsistencies introduced by these generative models with regard to the corresponding annotation data makes it unfeasible for validation.…”

Section: Visual Perception Datasetsmentioning

confidence: 99%

Training and Validation of Visual Perception Functions for Autonomous Driving with Synthetic Data

Hagn

View full text Add to dashboard Cite

This work deals with the usability of synthetically generated data for training and validation of visual perception functions applied in autonomous driving. Synthetically generated images allow the creation of safety critical scenarios which are potentially dangerous to capture in the real-world and additionally deliver pixel perfect ground truth annotations. However, applying synthetic images to perception functions trained on real-world data poses the problem of bridging the domain gap. This is true for both training and validating with synthetic images. Therefore, the domain gap has to be sufficiently understood to generate synthetically images viable to be used for training and validation. A new domain discrepancy metric is introduced and applied to optimize the parameters of a realistic sensor simulation effectively reducing the domain gap. Several influence factors on the domain gap are disentangled. The visual detection impairing factors are introduced and shown to have a high influence on the detectability of pedestrians. Additionally, these factors are used to calibrate a weighting loss function to increase the perception performance on real-world pedestrians. New methods for perception validation are introduced. The deep variational data synthesis and the classification of visual detection impairing factors. While the former method searches for perception faults by parameterized probabilistic image creation, the latter method detects perception faults by the disagreement of a detectability classifier and the actual detection result. The findings of both training and validating were influencing the creation of two synthetic validation datasets, VALERIE and SynPeDS.

show abstract

Does Redundancy in AI Perception Systems Help to Test for Super-Human Automated Driving Performance?

Gottschalk

Rottmann

Maida

2022

Deep Neural Networks and Data for Automated Driving

View full text Add to dashboard Cite

While automated driving is often advertised with better-than-human driving performance, this chapter reviews that it is nearly impossible to provide direct statistical evidence on the system level that this is actually the case. The amount of labeled data needed would exceed dimensions of present-day technical and economical capabilities. A commonly used strategy therefore is the use of redundancy along with the proof of sufficient subsystems’ performances. As it is known, this strategy is efficient especially for the case of subsystems operating independently, i.e., the occurrence of errors is independent in a statistical sense. Here, we give some first considerations and experimental evidence that this strategy is not a free ride as the errors of neural networks fulfilling the same computer vision task, at least in some cases, show correlated occurrences of errors. This remains true, if training data, architecture, and training are kept separate or independence is trained using special loss functions. Using data from different sensors (realized by up to five 2D projections of the 3D MNIST dataset) in our experiments is more efficiently reducing correlations, however not to an extent that is realizing the potential of reduction of testing data that can be obtained for redundant and statistically independent subsystems.

show abstract

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Cited by 4 publications

References 20 publications

Semantic Concept Testing in Autonomous Driving by Extraction of Object-Level Annotations from CARLA

Semantic Concept Testing in Autonomous Driving by Extraction of Object-Level Annotations from CARLA

Training and Validation of Visual Perception Functions for Autonomous Driving with Synthetic Data

Does Redundancy in AI Perception Systems Help to Test for Super-Human Automated Driving Performance?

Contact Info

Product

Resources

About