Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Abstract

Manufacturers of safety-critical systems must make the case that their product is sufficiently safe for public deployment. Much of this case often relies upon critical event outcomes from real-world testing, requiring manufacturers to be strategic about how they allocate testing resources in order to maximize their chances of demonstrating system safety. This work frames the partially observable and belief-dependent problem of test scheduling as a Markov decision process, which can be solved efficiently to yield closed-loop manufacturer testing policies. By solving for policies over a wide range of problem formulations, we are able to provide high-level guidance for manufacturers and regulators on issues relating to the testing of safety-critical systems. This guidance spans an array of topics, including circumstances under which manufacturers should continue testing despite observed incidents, when manufacturers should test aggressively, and when regulators should increase or reduce the real-world testing requirements for safety-critical systems.

Test Your Self-Driving Algorithm: An Overview of Publicly Available Driving Datasets and Virtual Testing Environments

“…Future work in this area should evaluate what combination of publicly available datasets and virtual testing environments will result in the best fidelity level in terms of resulting performance in reality compared to what has been achieved with open-loop and closed-loop testing. Recent approaches combining dataset collection and virtual environment usage [56] and [57] have been observed encouraging future work. Furthermore, commonalities between existing datasets could be studied in greater details to find overlapping or complementary parts; in that regard, a standardized representation or encoding of all datasets could be proposed to enable a simplified comparison of a system-undertest using various datasets.…”

Simulating Emergent Properties of Human Driving Behavior Using Multi-Agent Reward Augmented Imitation Learning

“…The autonomous driving literature has established that it is infeasible to build a statistically significant case for the safety of a system solely through real-world testing [3], [4]. Validation through simulation is an alternative to real-world testing, with the ability to evaluate vehicle performance in large numbers of scenes quickly, safely, and economically [5]. This paper seeks to extend state of the art imitation learning to improve our ability to accurately generate realistic driving scenarios.…”