Application of Evolutionary Algorithms and Criticality Metrics for the Verification and Validation of Automated Driving Systems at Urban Intersections

Bussler, Andreas; Hartjen, Lukas; Philipp, Robin; Schuldt, Fabian

doi:10.1109/iv47402.2020.9304662

Cited by 23 publications

(11 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…III-A.2), we can extend the input space coverage not only locally and allow for completely new (controllable) scenarios to be generated and tested against. This is in line with the literature about scenario-based testing in the context of AD [12], [13].…”

Section: A Testing Of Machine Learning Modelssupporting

confidence: 91%

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Rosenzweig¹,

Brito²,

Kobialka³

et al. 2021

Preprint

View full text Add to dashboard Cite

Many machine learning applications can benefit from simulated data for systematic validation -in particular if real-life data is difficult to obtain or annotate. However, since simulations are prone to domain shift w.r.t. real-life data, it is crucial to verify the transferability of the obtained results.We propose a novel framework consisting of a generative label-to-image synthesis model together with different transferability measures to inspect to what extent we can transfer testing results of semantic segmentation models from synthetic data to equivalent real-life data. With slight modifications, our approach is extendable to, e.g., general multi-class classification tasks. Grounded on the transferability analysis, our approach additionally allows for extensive testing by incorporating controlled simulations. We validate our approach empirically on a semantic segmentation task on driving scenes. Transferability is tested using correlation analysis of IoU and a learned discriminator. Although the latter can distinguish between reallife and synthetic tests, in the former we observe surprisingly strong correlations of 0.7 for both cars and pedestrians.

show abstract

Section: A Testing Of Machine Learning Modelssupporting

confidence: 91%

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Rosenzweig¹,

Brito²,

Kobialka³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Parameters of control for activated behavior control influence an object's activation condition and behavior after being activated. Typical parameters grouped by the type of objects are: (i) NPC vehicles: velocity after certain simulation time [71]; trigger distance for activation [62], [68]; trigger distance for behavior change [65]; velocity after triggered [62], [65]; velocity [57], [67]; and lane change [57], [67] at each time interval.…”

Section: Control For Activated Behaviormentioning

confidence: 99%

“…(ii) NPC pedestrian: trigger distance for activation [60], [62], [65], [66]; velocity after triggered [62], [65], [66].…”

Section: Control For Activated Behaviormentioning

confidence: 99%

“…A scenario is considered critical if a safety measure is below a user-specified threshold. Two of the most commonly used measures are the minimum distance between the ego car and any other objects [43], [46], [47], [48], [53], [54], [55], [57], [59], [60], [62], [65], [66], [67], [68], [69], [70], [72], [73], [74] during the simulation (denoted as min distance) and minimum Time To Collision [100] (or its variants) between the ego car and any of other objects [43], [45], [51], [52], [61], [62], [63] during the simulation (denoted as min TTC). Many works (e.g.…”

Section: Misbehavior Propertymentioning

confidence: 99%

“…Algorithm Evaluation Metric is usually the minimum value itself e.g. min distance/min TTC [52], [62]. In the case of multiple objectives, hypervolume [43], [45], generational distance [45], and spread [45] have been used.…”

Section: Scenarios With the Best Valuesmentioning

confidence: 99%

See 2 more Smart Citations

A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation

Zhong¹,

Tang²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Automated Driving Systems (ADSS) have seen rapid progress in recent years. To ensure the safety and reliability of these systems, extensive testings are being conducted before their future mass deployment. Testing the system on the road is the closest to real-world and desirable approach, but it is incredibly costly. Also, it is infeasible to cover rare corner cases using such real-world testing. Thus, a popular alternative is to evaluate an ADS's performance in some well-designed challenging scenarios, a.k.a. scenario-based testing. High-fidelity simulators have been widely used in this setting to maximize flexibility and convenience in testing what-if scenarios. Although many works have been proposed offering diverse frameworks/methods for testing specific systems, the comparisons and connections among these works are still missing. To bridge this gap, in this work, we provide a generic formulation of scenario-based testing in high-fidelity simulation and conduct a literature review on the existing works. We further compare them and present the open challenges as well as potential future research directions.

show abstract

Simulation-Based Elicitation of Accuracy Requirements for the Environmental Perception of Autonomous Vehicles

Philipp

Qian

Hartjen

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Application of Evolutionary Algorithms and Criticality Metrics for the Verification and Validation of Automated Driving Systems at Urban Intersections

Cited by 23 publications

References 17 publications

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation

Simulation-Based Elicitation of Accuracy Requirements for the Environmental Perception of Autonomous Vehicles

Contact Info

Product

Resources

About