Risk-sensitive safety analysis using Conditional Value-at-Risk

Abstract

This paper develops a safety analysis method for stochastic systems that is sensitive to the possibility and severity of rare harmful outcomes. We define risksensitive safe sets as sub-level sets of the solution to a non-standard optimal control problem, where a random maximum cost is assessed using the Conditional Value-at-Risk (CVaR) functional. The solution to the control problem represents the maximum extent of constraint violation of the state trajectory, averaged over the α • 100% worst cases, where α ∈ (0, 1]. This problem is well-motivated but difficult to solve in a tractable fashion because temporal decompositions for risk functionals generally depend on the history of the system's behavior. Our primary theoretical contribution is to derive under-approximations to risksensitive safe sets, which are computationally tractable. Our method provides a novel, theoretically guaranteed, parameter-dependent upper bound to the CVaR of a maximum cost without the need to augment the state space. For a fixed parameter value, the solution to only one Markov decision process problem is required to obtain the underapproximations for any family of risk-sensitivity levels. In addition, we propose a second definition for risk-sensitive safe sets and provide a tractable method for their estimation without using a parameter-dependent upper bound. The second definition is expressed in terms of a new coher-

Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control

“…In prior work, we proposed the problem of optimizing the CVaR of a maximum cost to assess the magnitude of an MDP's constraint violation in a fraction of worst cases [17,18]. A classical robust safety analysis method [57], advancements in the theory of optimizing CVaR criteria for MDPs [43], and the practical importance of including an assessment of consequences in risk analysis inspired us to formulate this problem.…”

“…We proposed a DP-based method using state-space augmentation that provides an exact solution in principle for finitetime MDPs with Borel state and action spaces [18]. In addition, we devised a theoretically-guaranteed under-approximation method [17], which does not require state-space augmentation and thus is considerably more efficient, at the expense of being less accurate, in comparison to [18].…”

“…In 2002, Mihatsch and Neuneier proposed Q-learning and temporal difference learning algorithms that encode risk-aware preferences for MDPs with finitely many states and actions [114]. 17 They modified the standard algorithms using a specific weighting function and showed that the modified versions satisfy certain convergence properties [114]. At the same time, Borkar independently proposed a risk-aware Q-learning algorithm for finite-state finite-action MDPs that uses the Exponential-Utility functional to assess risk [80] (Sec.…”

A Risk-Averse Preview-based $Q$-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles

“…∀(s, a, a ) ∈ S k × A × A. This implies that Q * k is a feasible solution to (21). Also, for any Q ∈ C(S k , A) in a feasible region of ( 21), the inequality…”

“…Based on Proposition 3, this indicates that Q is a lower bound to Q * k . Since we are maximizing the objective function in (21), this means that Q * k is a maximizer of the optimization, as well as all bounded functions that differ from Q * k on a subset of S k × A × A with measure zero. This completes the proof.…”