Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning

Abstract


 
 
 Finding the most likely path to a set of failure states is important to the analysis of safety-critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due to the complex stochastic environment in which the system operates. As a result, safety validation is not only concerned about whether a failure can occur, but also discovering which failures are most likely to occur. This article presents adaptive stress testing (AST), a framework for finding the most likely path to a failure event in simulation. We consider a general black box setting for partially observable and continuous-valued systems operating in an environment with stochastic disturbances. We formulate the problem as a Markov decision process and use reinforcement learning to optimize it. The approach is simulation-based and does not require internal knowledge of the system, making it suitable for black-box testing of large systems. We present different formulations depending on whether the state is fully observable or partially observable. In the latter case, we present a modified Monte Carlo tree search algorithm that only requires access to the pseudorandom number generator of the simulator to overcome partial observability. We also present an extension of the framework, called differential adaptive stress testing (DAST), that can find failures that occur in one system but not in another. This type of differential analysis is useful in applications such as regression testing, where we are concerned with finding areas of relative weakness compared to a baseline. We demonstrate the effectiveness of the approach on an aircraft collision avoidance application, where a prototype aircraft collision avoidance system is stress tested to find the most likely scenarios of near mid-air collision.
 
 

A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems

“…The design of the cost function c is specific to the application, but can be done adhoc or using a formal measure of the satisfaction of a safety property. For simple safety proprieties such as collision avoidance, the miss distance (point of closest approach between agents) is a common choice (Koren et al, 2018;Lee et al, 2020). Balkan et al (2017) propose several possible functions for finding non-convergence behaviors in control systems, including Lyapunov-like functions, M -step Lyapunov-like functions, neural networks, and support vector machines.…”

“…Upper and lower bounds on robustness can be computed for incomplete trajectory segments (Dreossi et al, 2015). An approach for most-likely failure analysis called adaptive stress testing (AST) Lee et al, 2020) (11) where the specification ψ is for the system to avoid reaching a set of failure states S fail , and λ penalizes disturbances that do not end in a failure state. The log probability of the disturbance is awarded at each time step to encourage the discovery of likely failures.…”

“…Monte Carlo tree search (MCTS) is an online planning algorithm for sequential decision making that has seen success for long time-horizon problems (Silver et al, 2016), making it useful for safety validation (Delmas et al, 2019;Julian et al, 2020;Lee et al, 2020;Moss et al, 2020;Wicker et al, 2018;Zhang et al, 2018). Monte Carlo tree search uses online planning to determine the best actions to take from a starting state to maximize reward.…”

“…Delmas et al (2019) discretize the disturbance space into a small number of discrete disturbances that are representative of the continuous space. As an alternative to discretization, when adding a new node to the tree, Lee et al (2020) sample a new disturbance x t from a known distribution over disturbances by selecting a random seed uniformly at random and using it to produce disturbances from p(x). 2018) combined MCTS with global optimization, using MCTS for exploration of the disturbance trajectory space and global optimization for refinement of the disturbance trajectories.…”

“…There are several examples of applications to collision avoidance problems for aircraft. Lee et al (2020) validate the next-generation aircraft collision avoidance system (ACAS X), which makes recommendations to pilots to avoid mid-air collisions.…”