Efficient statistical validation with edge cases to evaluate Highly Automated Vehicles

Critical Scenario Identification for Realistic Testing of Autonomous Driving Systems

“…In this paper, we extend an approach for critical test scenario identification for autonomous driving and have used it for testing real autonomous driving systems. We argue that testing all possible driving scenarios in real road traffic is impractical, as it is expensive, time-consuming, and may still not cover all the rare-occurring traffic situations [15,21], in contrast to Kalra et al who claim that millions or even billions of miles of driving test are required to demonstrate the reliability of an autonomous vehicle [46]. Instead, testing of autonomous driving functions must be based on a feasible number of test scenarios and focus on the most critical ones [4,7].…”

“…Among these surrogate indicators, TTC is used the most according to a review study by Aliaksei et al [18]. Safety metrics can also be extracted from industrial standards and used as the criticality indicators, for example ISO-15622 for adaptive cruise control [19], ISO-26262 for general automotive development and test [20], and Responsibility-Sensitive Safety (RSS) for autonomous vehicles [21]. A number of performance metrics including safety, functionality, mobility, and drivers' comfort are used for generating test scenarios for autonomous vehicles by Feng et al [20], and they use a combination of the maneuver challenge and exposure frequency as the indicator of critical scenarios.…”

“…Lastly, we also differentiate the concept of critical scenarios from corner case scenarios (also referred as edge cases). Karunakaran et al define an edge case as an unknown and unsafe scenario which is hard to predict during the test and can lead to severe result for the autonomous vehicle [21]. Since critical scenarios can either be known or unknown scenarios, we believe the edge cases are a subset of the critical scenarios that are of high interest for its identification and testing the autonomous driving systems.…”

“…The generative model gets a higher reward when a riskier scenario is generated. Another study that uses reinforcement learning is reported by Karunakaran et al [21] for automatically generating scenarios and optimizing the learning towards the worst-case scenarios with respect to the RSS safety metrics. A few other studies that employ deep learning techniques include Batsch et al [8] using Gaussian Processes to train and optimize the parameter selection towards the most critical scenarios on the performance boundary, and Jenkins et al [27] using a recurrent neural network to generate accident scenarios for testing the autonomous driving systems based on the in-vehicle and vehicle-to-infrastructure data generated from simulators.…”

“…Even though diverse techniques for extracting or generating critical scenarios based on real driving data have been studied, limitations can be observed and are described in these studies as well. A prerequisite of using such techniques is a data set that is comprehensive [19], whereas it is well known that collecting real driving data at scale is both time-consuming and expensive, but still does not guarantee to include all corner cases [21,28]. As highlighted by Hallerbach et al [16], the major drawback of using recorded data is the incompleteness of the data set, thus we have to understand how the data is acquired and how representative it is.…”