Safety-constrained reinforcement learning with a distributional safety critic

Abstract

Safety is critical to broadening the real-world use of reinforcement learning. Modeling the safety aspects using a safety-cost signal separate from the reward and bounding the expected safety-cost is becoming standard practice, since it avoids the problem of finding a good balance between safety and performance. However, it can be risky to set constraints only on the expectation neglecting the tail of the distribution, which might have prohibitively large values. In this paper, we propose a method called Worst-Case Soft Actor Critic for safe RL that approximates the distribution of accumulated safety-costs to achieve risk control. More specifically, a certain level of conditional Value-at-Risk from the distribution is regarded as a safety constraint, which guides the change of adaptive safety weights to achieve a trade-off between reward and safety. As a result, we can compute policies whose worst-case performance satisfies the constraints. We investigate two ways to estimate the safety-cost distribution, namely a Gaussian approximation and a quantile regression algorithm. On the one hand, the Gaussian approximation is simple and easy to implement, but may underestimate the safety cost, on the other hand, the quantile regression leads to a more conservative behavior. The empirical analysis shows that the quantile regression method achieves excellent results in complex safety-constrained environments, showing good risk control.

Subtask-masked curriculum learning for reinforcement learning with application to UAV maneuver decision-making

“…Our approach adopts the policy reuse strategy that directly leverages a guide policy to sample trajectories, which facilitates rapid adaptation to a new task (Rosman et al, 2016;Fernández and Veloso, 2006). This strategy leads to better initial trajectories and improves the jump-start by providing a strong initial point for the learning algorithm Yang et al, 2022). The guide policy can take the form of a rule-based policy, expert policy, or well-trained policy (Ayeelyan et al, 2022).…”

Distributional Safety Critic for Stochastic Latent Actor-Critic

“…These approaches mainly differ in terms of the way they parameterize the return distribution and the distance metric that is used to measure the difference between two distributions. In this work, we follow the authors of [Yang et al 2023] and utilize the implicit quantile network (IQN) [Dabney et al 2018a] to approximate the cost return distribution.…”

“…Worst-case soft actor-critic (WCSAC) [Yang et al 2023] is a soft actor-critic (SAC) [Haarnoja et al 2018a, Haarnoja et al 2018b] based algorithm that uses a distributional safety critic to produce risk-averse behavior. To this end, the upper tail of the estimated distribution is used.…”

“…Our reasons for adopting a distributional perspective on the safety critic are twofold. First, inspired by the authors of [Yang et al 2023], we believe that it is particularly interesting to be able to produce risk-averse behavior in the context of safe reinforcement learning. Second, the use of a distributional safety critic can increase the accuracy of the predictions of cost returns, which, in turn, allows the agent to more accurately trade-off reward and cost in the CMDP setting.…”

Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator System

“…Apart from supervised and self-supervised learning, approaches such as meta-learning (Kirsch, van Steenkiste, and Schmidhuber 2020;Guo, Wu, and Lee 2022), transfer learning (Guo et al 2019;Vrbančič and Podgorelec 2020) and curriculum learning (Bengio et al 2009;Park and Park 2022;Hu et al 2022) have also demonstrated their capacity to adapt pre-trained policies to novel environments through retraining (Packer et al 2019). Integration of techniques such safe RL is recommended within the training process to allow the agent to avoid hazardous states, thereby ensuring consistent normal performance and enhancing training efficiency (Yang et al 2021(Yang et al , 2022. When confronted with expanded state space, the latter category of solutions that retrain learned policies is often favored.…”