Validation of Perception and Decision-Making Systems for Autonomous Driving via Statistical Model Checking

Barbier, Mathieu; Renzaglia, Alessandro; Quilbeuf, Jean; Rummelhard, Lukas; Paigwar, Anshul; Laugier, Christian; Legay, Axel; Ibañez‐Guzmán, Javier; Simonin, Olivier

doi:10.1109/ivs.2019.8813793

Cited by 26 publications

(15 citation statements)

References 28 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Testing on simulated raw data allows the perception algorithm to be tested without the need for physical sensor hardware (e.g., [169] for grid mapping). For testing object perception algorithms, a simulation environment such as Ref.…”

Section: Uncertainty Removalmentioning

confidence: 99%

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

2022

View full text Add to dashboard Cite

Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. The paper focuses on testing for verification and validation purposes at the interface between perception and planning, and structures the analysis along the three axes (1) test criteria and metrics, (2) test scenarios, and (3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. It is found that the realization of safety-oriented perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.

show abstract

Section: Uncertainty Removalmentioning

confidence: 99%

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

2022

View full text Add to dashboard Cite

show abstract

“…[1], [2], to argue for the safety of ML models. In statistical model checking [11], ML models are validated using Key Performance Indicators of interest in combination with statistics and modeling the ML components as probabilistic systems. In this paper and in many other testing approaches, simulation-based testing, e.g., [3]- [5], is deployed in which simulators are used to create testing data (cf.…”

Section: A Testing Of Machine Learning Modelsmentioning

confidence: 99%

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Rosenzweig¹,

Brito²,

Kobialka³

et al. 2021

Preprint

View full text Add to dashboard Cite

Many machine learning applications can benefit from simulated data for systematic validation -in particular if real-life data is difficult to obtain or annotate. However, since simulations are prone to domain shift w.r.t. real-life data, it is crucial to verify the transferability of the obtained results.We propose a novel framework consisting of a generative label-to-image synthesis model together with different transferability measures to inspect to what extent we can transfer testing results of semantic segmentation models from synthetic data to equivalent real-life data. With slight modifications, our approach is extendable to, e.g., general multi-class classification tasks. Grounded on the transferability analysis, our approach additionally allows for extensive testing by incorporating controlled simulations. We validate our approach empirically on a semantic segmentation task on driving scenes. Transferability is tested using correlation analysis of IoU and a learned discriminator. Although the latter can distinguish between reallife and synthetic tests, in the former we observe surprisingly strong correlations of 0.7 for both cars and pedestrians.

show abstract

“…A statistical model checking approach to validate the collision risk assessment generated by a probabilistic perception system is proposed in [7]. Validation of perception and decision-making systems for autonomous driving via statistical model checking is investigated in [8]. A machine learning-based approach for uncertainty modeling and runtime verification of autonomous vehicles driving control is proposed in [9].…”

Section: Introductionmentioning

confidence: 99%

Formal Estimation of Collision Risks for Autonomous Vehicles: A Compositional Data-Driven Approach

Lavaei¹,

Lillo²,

Censi³

2021

Preprint

View full text Add to dashboard Cite

In this work, we propose a compositional data-driven approach for the formal estimation of collision risks for autonomous vehicles (AVs) with black-box dynamics acting in a stochastic multi-agent framework. The proposed approach is based on the construction of sub-barrier certificates for each stochastic agent via a set of data collected from its trajectories while providing a-priori guaranteed confidence on the datadriven estimation. In our proposed setting, we first cast the original collision risk problem for each agent as a robust optimization program (ROP). Solving the acquired ROP is not tractable due to an unknown model that appears in one of its constraints. To tackle this difficulty, we collect finite numbers of data from trajectories of each agent and provide a scenario optimization program (SOP) corresponding to the original ROP. We then establish a probabilistic bridge between the optimal value of SOP and that of ROP, and accordingly, we formally construct the sub-barrier certificate for each unknown agent based on the number of data and a required level of confidence. We then propose a compositional technique based on small-gain reasoning to quantify the collision risk for multi-agent AVs with some desirable confidence based on sub-barrier certificates of individual agents constructed from data. For the case that the proposed compositionality conditions are not satisfied, we provide a relaxed version of compositional results without requiring any compositionality conditions but at the cost of providing a potentially conservative collision risk. Eventually, we develop our approaches for non-stochastic multi-agent AVs. We demonstrate the effectiveness of our proposed results by applying them to a vehicle platooning consisting of 100 vehicles with 1 leader and 99 followers. We formally estimate the collision risk for the whole network by collecting sampled data from trajectories of each agent.

show abstract

Validation of Perception and Decision-Making Systems for Autonomous Driving via Statistical Model Checking

Cited by 26 publications

References 28 publications

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

Formal Estimation of Collision Risks for Autonomous Vehicles: A Compositional Data-Driven Approach

Contact Info

Product

Resources

About