Safe multi-agent reinforcement learning for multi-robot control

Gu, Shangding; Kuba, Jakub Grudzien; Chen, Yuanpei; Du, Yali; Long, Yang; Knoll, Alois; Yang, Yaodong

doi:10.1016/j.artint.2023.103905

Cited by 30 publications

(11 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An overview of the recent advances in safe RL can be found in the latest surveys [16], [17]. In this section, we will focus on the prior works most related to our approach.…”

Section: Related Workmentioning

confidence: 99%

A Multiplicative Value Function for Safe and Efficient Reinforcement Learning

Bührer,

Zhang,

Liniger

et al. 2023

2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

An emerging field of sequential decision problems is safe Reinforcement Learning (RL), where the objective is to maximize the reward while obeying safety constraints. Being able to handle constraints is essential for deploying RL agents in real-world environments, where constraint violations can harm the agent and the environment. To this end, we propose a safe model-free RL algorithm with a novel multiplicative value function consisting of a safety critic and a reward critic. The safety critic predicts the probability of constraint violation and discounts the reward critic that only estimates constraint-free returns. By splitting responsibilities, we facilitate the learning task leading to increased sample efficiency. We integrate our approach into two popular RL algorithms, Proximal Policy Optimization and Soft Actor-Critic, and evaluate our method in four safety-focused environments, including classical RL benchmarks augmented with safety constraints and robot navigation tasks with images and raw Lidar scans as observations. Finally, we make the zero-shot sim-to-real transfer where a differential drive robot has to navigate through a cluttered room. Our code can be found at https://github.com/nikeke19/ Safe-Mult-RL.

show abstract

“…An overview of the recent advances in safe RL can be found in the latest surveys [16], [17]. In this section, we will focus on the prior works most related to our approach.…”

Section: Related Workmentioning

confidence: 99%

A Multiplicative Value Function for Safe and Efficient Reinforcement Learning

Bührer,

Zhang,

Liniger

et al. 2023

2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…Reinforcement Learning (RL) has demonstrated remarkable performance in various scenarios (Gu et al 2022b), such as the game of Go (Silver et al 2016), autonomous driving (Kiran et al 2021;Gu et al 2022a), and robotics (Kober, Bagnell, and Peters 2013;Gu et al 2023b). However, the majority of RL methods are restricted to simulation environments due to safety concerns associated with deploying RL in real-world settings.…”

Section: Introductionmentioning

confidence: 99%

Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation

Gu,

Sel,

Ding

et al. 2024

AAAI

View full text Add to dashboard Cite

Ensuring the safety of Reinforcement Learning (RL) is crucial for its deployment in real-world applications. Nevertheless, managing the trade-off between reward and safety during exploration presents a significant challenge. Improving reward performance through policy adjustments may adversely affect safety performance. In this study, we aim to address this conflicting relation by leveraging the theory of gradient manipulation. Initially, we analyze the conflict between reward and safety gradients. Subsequently, we tackle the balance between reward and safety optimization by proposing a soft switching policy optimization method, for which we provide convergence analysis. Based on our theoretical examination, we provide a safe RL framework to overcome the aforementioned challenge, and we develop a Safety-MuJoCo Benchmark to assess the performance of safe RL algorithms. Finally, we evaluate the effectiveness of our method on the Safety-MuJoCo Benchmark and a popular safe benchmark, Omnisafe. Experimental results demonstrate that our algorithms outperform several state-of-the-art baselines in terms of balancing reward and safety optimization.

show abstract

“…Safe reinforcement learning aims to obtain a rewardmaximizing policy within a constrained manifold (Garcıa & Fernández, 2015;Brunke et al, 2021), showing advantages to satisfy the safety requirements in real-world applications (Ray et al, 2019;Gu et al, 2022). However, most deep safe RL approaches focus on the safety during deployment, i.e., after training, while ignoring the constraint violation costs during training (Xu et al, 2022b).…”

Section: Introductionmentioning

confidence: 99%