Reachability-based safe learning with Gaussian processes

Abstract

Abstract-Reinforcement learning for robotic applications faces the challenge of constraint satisfaction, which currently impedes its application to safety critical systems. Recent approaches successfully introduce safety based on reachability analysis, determining a safe region of the state space where the system can operate. However, overly constraining the freedom of the system can negatively affect performance, while attempting to learn less conservative safety constraints might fail to preserve safety if the learned constraints are inaccurate. We propose a novel method that uses a principled approach to learn the system's unknown dynamics based on a Gaussian process model and iteratively approximates the maximal safe set. A modified control strategy based on real-time model validation preserves safety under weaker conditions than current approaches. Our framework further incorporates safety into the reinforcement learning performance metric, allowing a better integration of safety and learning. We demonstrate our algorithm on simulations of a cart-pole system and on an experimental quadrotor application and show how our proposed scheme succeeds in preserving safety where current approaches fail to avoid an unsafe condition.

Temporal-difference learning for online reachability analysis

“…Since the control u * S (w) drives the state w to f ∆t (w, u * S (w), d(w)) then ∃u(·) that can keep the state trajectory starting at w out of K for all time. 2 This leads to a contradiction, and w must indeed be a safe state. Furthermore, if V k+1 S (w) = 0 then (17) applies the control u * S (w), which keeps the state trajectory in the safe set.…”

“…However, more time will be needed to update mis-labeled safe states as safe. 2 Here we are assuming that if the state trajectory enters K in between samples then is stays there.…”

“…With exact knowledge of the dynamics, we can obtain the optimal safety value function, which represents the largest safe set of the system [2], and will also be referred to as the optimal safe set. Given an approximation of the optimal safety value function, the objective is to update the approximation such that the system performance regarding its other specifications approaches that attained when using the optimal safety value function while still ensuring constraintsatisfaction.…”

“…The authors of [2] and [11] propose iterating back and forth between inferring models of the system from collected data, and recomputing the allowable region based on the inferred model. Reachability analysis is computationally expensive as it scales exponentially with the dimension of the state space, so recomputing the allowable region multiple times is undesirable.…”

Safe robot execution in model-based reinforcement learning

“…These approaches either require a safety function to avoid dangerous states while exploring [6], [7], or use simulators to confirm the safety of states [8]. In our case, the decision maker does not have any prior information about the risky states that may be found, but in contrast, every time a dead-end is reached, it will learn the causes and avoid them in the future.…”