The stochastic reach-avoid problem and set characterization for diffusions

Abstract

Abstract. In this article we approach a class of stochastic reachability problems with state constraints from an optimal control perspective. Preceding approaches to solving these reachability problems are either confined to the deterministic setting or address almost-sure stochastic requirements. In contrast, we propose a methodology to tackle problems with less stringent requirements than almost sure. To this end, we first establish a connection between two distinct stochastic reach-avoid problems and three classes of stochastic optimal control problems involving discontinuous payoff functions. Subsequently, we focus on solutions of one of the classes of stochastic optimal control problems-the exit-time problem, which solves both the two reach-avoid problems mentioned above. We then derive a weak version of a dynamic programming principle (DPP) for the corresponding value function; in this direction our contribution compared to the existing literature is to develop techniques that admit discontinuous payoff functions. Moreover, based on our DPP, we provide an alternative characterization of the value function as a solution of a partial differential equation in the sense of discontinuous viscosity solutions, along with boundary conditions both in Dirichlet and viscosity senses. Theoretical justifications are also discussed to pave the way for deployment of off-the-shelf PDE solvers for numerical computations. Finally, we validate the performance of the proposed framework on the stochastic Zermelo navigation problem.

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process

“…A classical problem in control theory is to ensure that the state trajectory reaches a target set while avoiding an unsafe set. This is referred to as Reach-Avoid problem in [7], [8] and [9] and references therein. Many variants of this problem (in different settings) have been addressed in the literature.…”

Grid-Free Computation of Probabilistic Safety with Malliavin Calculus

“…[33] builds on [25] by employing Monte Carlo (MC) techniques for estimating probabilities of events, and [33] uses multilevel splitting (MLS), a variance-reduction technique that can improve both efficiency and accuracy. Again over SHS, [30] establishes a connection between stochastic reachavoid problems -problems encompassing both reachability and safety, also known as constrained reachability problems -and optimal control problems involving discontinuous payoff functions. Focussing on a particular stochastic optimal control problem, the exit-time problem mentioned above, [30] provides its characterisation as a solution of a partial differential equation in the sense of viscosity solutions, along with Dirichlet boundary conditions.…”

“…Again over SHS, [30] establishes a connection between stochastic reachavoid problems -problems encompassing both reachability and safety, also known as constrained reachability problems -and optimal control problems involving discontinuous payoff functions. Focussing on a particular stochastic optimal control problem, the exit-time problem mentioned above, [30] provides its characterisation as a solution of a partial differential equation in the sense of viscosity solutions, along with Dirichlet boundary conditions. [38] establishes an optimisation scheme for computing probabilistic safety of SHS, combining the use of barrier certificates and of potential theory.…”

Sufficient Conditions for Robust Probabilistic Reach-Avoid-Stay Specifications using Stochastic Lyapunov-Barrier Functions

“…Works in [31], [35], [14] characterized value functions for reachability/reach-avoid problems in discrete-time continuous-state stochastic systems and applied dynamic programming for synthesizing optimal controllers. Authors in [9] developed a weak dynamic programming principle for the value functions of probabilistic reach-avoid specifications in continuous-time continuous-state stochastic systems, which provides compatibility for non-almost-sure probabilistic requirements.…”