Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

Fremont, Daniel J.; Kim, Edward; Pant, Yash Vardhan; Seshia, Sanjit A.; Acharya, Atul; Bruso, Xantha; Wells, Paul; Lemke, Steve; Lu, Qiang; Mehta, Shalin

doi:10.1109/itsc45102.2020.9294368

Cited by 120 publications

(58 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The typical scenario-based testing process generally includes at least three phases (cf. [11][12][13]29,30]): the identification of relevant scenarios, the modeling of the identified scenarios, and the subsequent simulative execution and evaluation of the modeled scenarios (cf. Figure 1).…”

Section: Holistic View On Scenario-based Simulationmentioning

confidence: 99%

Towards a Model-Based Multi-Layered Approach to Describe Traffic Scenarios on a Technical Level

Reiher

Hahn

2021

JMSE

View full text Add to dashboard Cite

Highly automated vehicles are increasingly gaining the public’s attention. To achieve broad acceptance for the deployment of such vehicles, it is necessary to ensure their functionality and safety. One approach that has become popular in research is the scenario-based approach. However, manual testing of such complex systems is impractical and time-consuming. Using simulations to run and evaluate such scenarios appears to be the most viable approach. This, in turn, raises new challenges, especially in modeling the scenarios to be tested simulatively and incorporating the system under test as part of these. Since existing solutions do not solve these challenges satisfactorily—due to the strict separation of scenario and simulation model, among other reasons—this work addresses the need for a standardized, holistic, and extensible approach for modeling traffic scenarios to be executed simulatively. Requirements for such an approach are identified with focus on its application in simulation- and scenario-based verification and validation. Based on these, a model-based multi-layered approach is proposed. The foundations of this are then implemented utilizing a Meta Object Facility based heavyweight extension of the Unified Modeling Language metamodel. The resulting metamodel is used to demonstrate the applicability of the proposed approach by modeling a maritime traffic scenario.

show abstract

Section: Holistic View On Scenario-based Simulationmentioning

confidence: 99%

Towards a Model-Based Multi-Layered Approach to Describe Traffic Scenarios on a Technical Level

Reiher

Hahn

2021

JMSE

View full text Add to dashboard Cite

show abstract

“…For instance, Yamaguchi et al [266] present a series of simulations to combine requirement mining and model checking in simulation-based testing for end-to-end systems. Fremont et al [59] present a scenariobased testing tool which generates test cases by combining specification of possible scenarios and safety properties. In many cases, samples and scenarios from the simulation tests are used to improve the training set.…”

Section: Formal Testingmentioning

confidence: 99%

“…However, it is challenging to mitigate the gap between simulation and real-world situations, causing questionable transfer of simulated verification and testing results. Recent work starts exploring how simulated formal simulation aid in designing real-world tests [59]. Additionally, thorough and scenario-based simulations enable system verification in broader terms such as monitoring interactions between ML modules in a complex system, e.g., how would an attack on the perception module affect the control module.…”

Section: Inherently Safe and Transparent Designmentioning

confidence: 99%

Taxonomy of Machine Learning Safety: A Survey and Primer

Mohseni¹,

Wang²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

The open-world deployment of Machine Learning (ML) algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities such as interpretability, verifiability, and performance limitations. Research explores different approaches to improve ML dependability by proposing new models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. In this paper, we review and organize practical ML techniques that can improve the safety and dependability of ML algorithms and therefore ML-based software. Our organization maps state-of-the-art ML techniques to safety strategies in order to enhance the dependability of the ML algorithm from different aspects, and discuss research gaps as well as promising solutions.

show abstract

“…Transfer learning has been one of the most popular topics in robotics, especially since machine learning techniques have become widely exploited. The idea behind transfer learning is to migrate the knowledge between similar problems to boost the training process [6], take advantage of existing knowledge [7], and reduce the risk of training [8], [9]. Although machine learning approaches have been massively explored, we cannot ignore that they typically require a large amount of data and a lot of effort in training the model.…”

Section: Related Workmentioning

confidence: 99%

A Conformal Mapping-based Framework for Robot-to-Robot and Sim-to-Real Transfer Learning

Gao¹,

Bezzo²

2021

Preprint

View full text Add to dashboard Cite

This paper presents a novel method for transferring motion planning and control policies between a teacher and a learner robot. With this work, we propose to reduce the sim-to-real gap, transfer knowledge designed for a specific system into a different robot, and compensate for system aging and failures. To solve this problem we introduce a Schwarz-Christoffel mapping-based method to geometrically stretch and fit the control inputs from the teacher into the learner command space. We also propose a method based on primitive motion generation to create motion plans and control inputs compatible with the learner's capabilities. Our approach is validated with simulations and experiments with different robotic systems navigating occluding environments.

show abstract

Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

Cited by 120 publications

References 16 publications

Towards a Model-Based Multi-Layered Approach to Describe Traffic Scenarios on a Technical Level

Towards a Model-Based Multi-Layered Approach to Describe Traffic Scenarios on a Technical Level

Taxonomy of Machine Learning Safety: A Survey and Primer

A Conformal Mapping-based Framework for Robot-to-Robot and Sim-to-Real Transfer Learning

Contact Info

Product

Resources

About