Test Framework for Automatic Test Case Generation and Execution Aimed at Developing Trustworthy AVs from Both Verifiability and Certifiability Aspects

Zhang, Xizhe; Khastgir, Siddartha; Asgari, Hamid; Jennings, Paul

doi:10.1109/itsc48978.2021.9564542

Cited by 10 publications

(20 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the most recent publications [23], [24], the major focus has been on formalization of scenario-based verification and validation. Zhang et al [23] provide a test framework that consist of test scenarios, different types of testing and allocation of tests, generation of test cases, data collection as well as analysis and correlation of obtained results. Scholtes et al [24] focuses on structured modeling of urban road environment and traffic using a six-layer model approach.…”

Section: Related Workmentioning

confidence: 99%

Discovery of Perception Performance Limiting Triggering Conditions in Automated Driving

Adee

Gansch

Liggesmeyer

et al. 2021

2021 5th International Conference on System Reliability and Safety (ICSRS)

View full text Add to dashboard Cite

Highly automated driving (HAD) vehicles are complex systems operating in an open context. Performance limitations originating from sensing and understanding the open context under triggering conditions may result in unsafe behavior, thus, need to be identified and modeled. This aspect of safety is also discussed in standardization activities such as ISO 21448, safety of the intended functionality (SOTIF). Although SOTIF provides a non-exhaustive list of scenario factors to identify and analyze performance limitations under triggering conditions, no concrete methodology is yet provided to identify novel triggering conditions.We propose a methodology to identify and model novel triggering conditions in a scene in order to assess SOTIF using Bayesian network (BN) and p-value hypothesis testing. The experts provide the initial BN structure while the conditional belief tables (CBTs) are learned using dataset. P-value hypothesis testing is used to identify the relevant subset of scenes. These scenes are then analyzed by experts who provide potential triggering conditions present in the scenes. The novel triggering conditions are modeled in the BN and retested. As a case study, we provide p-value hypothesis testing of BN of LIDAR using real world data.

show abstract

Section: Related Workmentioning

confidence: 99%

Discovery of Perception Performance Limiting Triggering Conditions in Automated Driving

Adee

Gansch

Liggesmeyer

et al. 2021

2021 5th International Conference on System Reliability and Safety (ICSRS)

View full text Add to dashboard Cite

show abstract

“…Emails: {antonio.bruto-da-costa, patrick.irvine, jason.zhang, s.khastgir.1, paul.jennings}@warwick.ac.uk quality of miles driven by the ADS, as opposed to the quantity of testing miles [4], [5], exposing the ADS to the myriad conditions it is expected to respond to. This is further asserted through various studies and the adoption of scenario-based testing as the basis of V&V approaches [6]- [11]. A scenario is a 'temporal development between several scenes in a sequence of scenes.…”

Section: Introductionmentioning

confidence: 99%

Ontology-based Scenario Generation for Automated Driving Systems Verification and Validation using Rules of the Road

da Costa,

Irvine,

Zhang

et al. 2024

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

The verification and validation (V&V) process for Automated Driving Systems (ADS) has undergone a significant transformation in defining the meaning of safety. Initially rooted in the quantity of miles driven, it has now shifted towards emphasizing the quality of test miles. These test miles must effectively capture the full spectrum of behaviours and operational design domains (ODD) of the ADS. To assess an ADS's compliance with specific rules or requirements, a connection must be established between the rules and the scenarios used for testing. In this paper, we propose a targeted scenario generation methodology aimed at testing ADS against formal rules. Our approach leverages ontologies to represent objects and their relationships in a scenario. The rules, are first formally specified, expressed as horn clauses. We then employ a rule transformation process, along with off-the-shelf reasoning tools, to generate corresponding scenarios. These generated scenarios may then utilized to test the ADS's adherence to the specified rules. To illustrate the effectiveness of our methodology, we present an application to example rules derived from both the UK Highway Code and the Vienna Conventions. By utilizing our approach, we enhance the precision and rigor of the verification and validation flow for ADS, ensuring improved safety measures during operation.Index Terms-scenario, scenario generation, ontologies, rules of the road, verification and validation, automated driving systems. 1) Methodology forFormalising Rules of the Road: We propose a comprehensive methodology for formalising rules of the road. To illustrate the practical application of this methodology, we utilise examples from the United Kingdom (UK) Highway Code [16] and the Vienna Convention [17]. Through these examples, we discussCopyright IEEE

show abstract

“…Other than a scene, a scenario spans a certain amount of time. ' Due to the widespread adoption of scenario-based testing as a part of the V&V approaches [7]- [11], many Scenario Description Languages(SDLs) have been developed. A SDL is a plain-text format for describing a scenario.…”

Section: Introductionmentioning

confidence: 99%

Writing Accessible and Correct Test Scenarios for Automated Driving Systems

Costa

Irvine

Zhang

et al. 2022

2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

Self Cite

View full text Add to dashboard Cite

For Automated Driving Systems (ADSs), vehicle safety and functional correctness are assessed against scenarios the ADS would encounter -within or outside its operational design domain (ODD). A scenario specifies conditions and events that an ADS is expected to respond to when deployed. Scenario specifications underpin the V&V life-cycle, and are used by a diverse set of stakeholders -from engineers to regulators. Due to the diversity in stakeholder expertise, scenarios must be available at different levels of detail, and further the chance of writing syntactically or semantically incorrect scenarios is high. Present-day Scenario Description Languages (SDLs) need to be supported by technologies to help authors compose scenarios and provide a mechanism for easy translation into executable forms for virtual or real-life testing. This paper addresses these issues by building on the existing two-level abstraction WMG-SDL in the following ways, (1) introducing a human-readable, natural language SDL, replacing the former Level-1 SDL, and complementing the more detailed Level-2 SDL, which is now syntax aligned with the ODD Taxonomy defined in ISO 34503, and (2) providing a tool consisting of a parser and a validator to assist writing syntactically and semantically correct scenarios. Our tool may be used within a graphical scenario editing interface or on the command-line. Further, for developers, an object-oriented interface for parsed scenarios enables further development and integration with off-the-shelf ADS simulation and language tools. The tools and technologies described in this paper are to be made open-source.

show abstract

Test Framework for Automatic Test Case Generation and Execution Aimed at Developing Trustworthy AVs from Both Verifiability and Certifiability Aspects

Abstract: How to cite:Please refer to published version for the most recent bibliographic citation information.

Cited by 10 publications

References 14 publications

Discovery of Perception Performance Limiting Triggering Conditions in Automated Driving

Discovery of Perception Performance Limiting Triggering Conditions in Automated Driving

Ontology-based Scenario Generation for Automated Driving Systems Verification and Validation using Rules of the Road

Writing Accessible and Correct Test Scenarios for Automated Driving Systems

Contact Info

Product

Resources

About