Automated scenario generation for regression testing of autonomous vehicles

Rocklage, Elias; Kraft, Heiko; Karatas, Abdullah; Seewig, Jörg

doi:10.1109/itsc.2017.8317919

Cited by 64 publications

(31 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An automated framework for regression testing is presented by [80], whereby parameter variations of roads, static and dynamic objects and also of environmental conditions are automatically created and combined. To make sure that all scenarios are physically reasonable, a modified In Parameter Order Generalized (IPOG) algorithm using a nonrecursive backtracking algorithm is used, in combination with a trajectory planner.…”

Section: A Sampling Within Parameter Rangesmentioning

confidence: 99%

See 1 more Smart Citation

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

When will automated vehicles come onto the market? This question has puzzled the automotive industry and society for years. The technology and its implementation have made rapid progress over the last decade, but the challenge of how to prove the safety of these systems has not yet been solved. Since a market launch without proof of safety would neither be accepted by society nor by legislators, much time and many resources have been invested into safety assessment in recent years in order to develop new approaches for an efficient assessment. This paper therefore provides an overview of various approaches, and gives a comprehensive survey of the so-called scenario-based approach. The scenario-based approach is a promising method, in which individual traffic situations are typically tested by means of virtual simulation. Since an infinite number of different scenarios can theoretically occur in real-world traffic, even the scenario-based approach leaves the question unanswered as to how to break these down into a finite set of scenarios, and find those which are representative in order to render testing more manageable. This paper provides a comprehensive literature review of related safety-assessment publications that deal precisely with this question. Therefore, this paper develops a novel taxonomy for the scenario-based approach, and classifies all literature sources. Based on this, the existing methods will be compared with each other and, as one conclusion, the alternative concept of formal verification will be combined with the scenario-based approach. Finally, future research priorities are derived.

show abstract

Section: A Sampling Within Parameter Rangesmentioning

confidence: 99%

“…To make sure that all scenarios are physically reasonable, a modified In Parameter Order Generalized (IPOG) algorithm using a nonrecursive backtracking algorithm is used, in combination with a trajectory planner. According to [80], the algorithm works as intended, but needs to become more efficient in the future.…”

Section: A Sampling Within Parameter Rangesmentioning

confidence: 99%

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Previous work [89] has shown that optimization methods can be applied to generate scenarios by maximizing a scenario quality metric. Scenario generation has been applied extensively to evaluating autonomous vehicles [6,65,2,76,29,77]. Contrary to model-checking and formal methods [12,68], which require a model describing the system's performance such as a finite-state machine [60] or process algebras [68], black-box approaches do not require access to a model.…”

Section: Problem Statementmentioning

confidence: 99%

A Quality Diversity Approach to Automatically Generating Human-Robot Interaction Scenarios in Shared Autonomy

Fontaine¹,

Nikolaidis²

2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring diverse scenarios of humans and robots interacting in simulation can improve understanding of the robotic system and avoid potentially costly failures in realworld settings. We formulate this problem as a quality diversity (QD) problem, where the goal is to discover diverse failure scenarios by simultaneously exploring both environments and human actions. We focus on the shared autonomy domain, where the robot attempts to infer the goal of a human operator, and adopt the QD algorithm MAP-Elites to generate scenarios for two published algorithms in this domain: shared autonomy via hindsight optimization and linear policy blending. Some of the generated scenarios confirm previous theoretical findings, while others are surprising and bring about a new understanding of state-of-the-art implementations. Our experiments show that MAP-Elites outperforms Monte-Carlo simulation and optimization based methods in effectively searching the scenario space, highlighting its promise for automatic evaluation of algorithms in human-robot interaction.

show abstract

“…We illustrate various challenges of model generation and state space exploration in the context of critical traffic scenarios for autonomous driving. Specifically, we aim at generating instances of traffic scenarios where the vision of the vehicle-under-test (referred to as the ego-vehicle [25,60]) is obstructed by the presence of other actors. Such scenarios have been identified as key challenges for the development of autonomous vehicle safety 1 .…”

Section: The Crossing Scenario Domainmentioning

confidence: 99%

“…Paracosm [42] applies Halton sampling on the parameter space to generate scenarios according to coverage criteria. The approach proposed in [60] combines combinatorial interaction testing, backtracking and motion planning to generate test cases for regression testing of autonomous vehicles. The authors of [16] propose a weighted search-based approach to find test scenarios with avoidable collisions.…”

Section: Related Workmentioning

confidence: 99%

Automated generation of consistent models using qualitative abstractions and exploration strategies

Babikian

Semeráth

et al. 2021

Softw Syst Model

View full text Add to dashboard Cite

Automatically synthesizing consistent models is a key prerequisite for many testing scenarios in autonomous driving to ensure a designated coverage of critical corner cases. An inconsistent model is irrelevant as a test case (e.g., false positive); thus, each synthetic model needs to simultaneously satisfy various structural and attribute constraints, which includes complex geometric constraints for traffic scenarios. While different logic solvers or dedicated graph solvers have recently been developed, they fail to handle either structural or attribute constraints in a scalable way. In the current paper, we combine a structural graph solver that uses partial models with an SMT-solver and a quadratic solver to automatically derive models which simultaneously fulfill structural and numeric constraints, while key theoretical properties of model generation like completeness or diversity are still ensured. This necessitates a sophisticated bidirectional interaction between different solvers which carry out consistency checks, decision, unit propagation, concretization steps. Additionally, we introduce custom exploration strategies to speed up model generation. We evaluate the scalability and diversity of our approach, as well as the influence of customizations, in the context of four complex case studies.

show abstract

Automated scenario generation for regression testing of autonomous vehicles

Cited by 64 publications

References 10 publications

Survey on Scenario-Based Safety Assessment of Automated Vehicles

Survey on Scenario-Based Safety Assessment of Automated Vehicles

A Quality Diversity Approach to Automatically Generating Human-Robot Interaction Scenarios in Shared Autonomy

Automated generation of consistent models using qualitative abstractions and exploration strategies

Contact Info

Product

Resources

About