Intersection Focused Situation Coverage-Based Verification and Validation Framework for Autonomous Vehicles Implemented in CARLA

Abstract

Autonomous Vehicles (AVs) i.e., self-driving cars, operate in a safetycritical domain, since errors in the autonomous driving software can lead to huge losses. Statistically, road intersections which are a part of the AVs operational design domain (ODD), have some of the highest accident rates. Hence, testing AVs to the limits on road intersections and assuring their safety on road intersections is pertinent, and thus the focus of this paper. We present a situation coverage-based (SitCov) AV-testing framework for the verification and validation (V&V) and safety assurance of AVs, developed in an open-source AV simulator named CARLA. The SitCov AV-testing framework focuses on vehicle-to-vehicle (V2V) interaction on a road intersection under different environmental conditions and intersection configuration situations (start/goal locations), using situation coverage criteria for automatic test suite generation for safety assurance of AVs. We have developed an ontology for intersection situations, and used it to generate a situation hyperspace i.e., the space of all possible situations arising from that ontology. For the evaluation of our SitCov AV-testing framework, we have seeded multiple faults in our ego AV, and compared situation coveragebased and random situation generation. We have found that both generation methodologies trigger around the same number of seeded faults, but the situation coverage-based generation tells us a lot more about the weaknesses of the autonomous driving algorithm of our ego AV, especially in edge-cases. Our code is publicly available online and since the simulation software (CARLA) is opensource, anyone can use our SitCov AV-testing framework and use it or build further on top of it. This paper aims to contribute to the domain of V&V and development of AVs, not only from a theoretical point of view, but also from the viewpoint of an open-source software contribution and releasing a flexible/effective tool for V&V and development of AVs.

One Ontology to Rule Them All: Corner Case Scenarios for Autonomous Driving

“…They build upon their previous works [40,45,28]. Tahir and Alexander [39] propose an ontology that focuses on intersections due to their high collision rates. They show that their scenarios can be executed in simulation, while focusing on changing weather conditions.…”

“…Ontology available Fuchs et al [16] 2008 --✓ ---Hummel [25] 2010 -✓ ----Hülsen et al [26] 2011 ✓ ✓ ----Armand et al [1] 2014 ------Zhao et al [47] 2017 -✓ ---✓ Bagschik et al [5] 2018 ✓ -----Chen and Kloul [13] 2018 ✓ -----Huang et al [24] 2019 ✓ -----Menzel et al [30] 2019 ✓ ✓ -✓ --Li et al [29] 2020 ✓ ✓ -✓ --Tahir and Alexander [39] 2022 ✓ --✓ --Hermann et al [23] 2022…”

Systematic Situation Coverage versus Random Situation Coverage for Safety Testing in an Autonomous Car Simulation

“…Hawkins et.al [11] and Tahir and Alexander [18] evaluated situation coverage approach that is łCan a situation generation testing method based on situation coverage approach outperform a randomly generation one in terms of fault detection?ž . Though the task environment of [11] and [18] were different with different seeded faults, they got almost same result.…”

“…Hawkins et.al [11] and Tahir and Alexander [18] evaluated situation coverage approach that is łCan a situation generation testing method based on situation coverage approach outperform a randomly generation one in terms of fault detection?ž . Though the task environment of [11] and [18] were different with different seeded faults, they got almost same result. They concluded that the situation coverage does not achieve significant benefits over random situation generation but coverage directed generation does cover a greater proportion of the situation space in the same amount of compute time.…”

“…In this paper, we will use slightly different fault set on Tahir et.al. [18] approach and test whether situation coverage based search approach does add additional value to random generation or not. From Tahir's work, it appeared that situation coverage was less effective than random testing in finding seeded faults.…”