Exploring Safer Behaviors for Deep Reinforcement Learning

Abstract

We consider Reinforcement Learning (RL) problems where an agent attempts to maximize a reward signal while minimizing a cost function that models unsafe behaviors. Such formalization is addressed in the literature using constrained optimization on the cost, limiting the exploration and leading to a significant trade-off between cost and reward. In contrast, we propose a Safety-Oriented Search that complements Deep RL algorithms to bias the policy toward safety within an evolutionary cost optimization. We leverage evolutionary exploration benefits to design a novel concept of safe mutations that use visited unsafe states to explore safer actions. We further characterize the behaviors of the policies over desired specifics with a sample-based bound estimation, which makes prior verification analysis tractable in the training loop. Hence, driving the learning process towards safer regions of the policy space. Empirical evidence on the Safety Gym benchmark shows that we successfully avoid drawbacks on the return while improving the safety of the policy.

Constrained Reinforcement Learning for Robotics via Scenario-Based Programming

“…Constrained reinforcement learning is an emerging field [13,14,12]. To show the effectiveness of our approach, we also compared it to an implementation of Lagrangian-PPO, as suggested by [15].…”

“…An emerging family of approaches for achieving these two goals, known as constrained DRL [12], attempts to simultaneously optimize two functions: the reward, which encodes the main objective of the task; and the cost, which represents the safety constraints. Current state-of-the-art algorithms include IPO [13], SOS [14], CPO [12], and Lagrangian approaches [15]. Despite their success in some applications, these methods generally suffer from significant setbacks: (i) there is no uniform and human-readable way of defining the required safety constraints; (ii) it is unclear how to encode these constraints as a signal for the training algorithm; and (iii) there is no clear method for balancing cost and reward during training, and thus there is a risk of producing sub-optimal policies.…”

“…While common DRL-training techniques were shown to give rise to high-performance policies for this task [20], these policies are often unsafe, inefficient, or unpredictable, thus dramatically limiting their potential deployment in real-world systems [21,14].…”

Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation

“…In particular, Safe DRL problems are typically modeled using Constrained Markov Decision Processes (CMPDs) [9], where an agent aims at maximizing a reward signal while keeping cost values accumulated upon visiting unsafe states under a hardcoded threshold. However, the constraints imposed by these approaches hinder exploration, failing to learn safe behaviors in complex environments [10], [11]. Alternative ways have been investigated to overcome the difficulty of designing Safe DRL algorithms that combine the concept of risk in the optimization while avoiding unsafe situations [8], [12], [13].…”

“…To this end, [11], [15] proposed a sample-based approximation method to enumerate the number of states in the space x that violate a specific property. Such a value referred to as violation, has been used to induce safety information during the training.…”