Reinforcement Learning by Guided Safe Exploration

Abstract

Safety is critical to broadening the application of reinforcement learning (RL). Often, we train RL agents in a controlled environment, such as a laboratory, before deploying them in the real world. However, the real-world target task might be unknown prior to deployment. Reward-free RL trains an agent without the reward to adapt quickly once the reward is revealed. We consider the constrained reward-free setting, where an agent (the guide) learns to explore safely without the reward signal. This agent is trained in a controlled environment, which allows unsafe interactions and still provides the safety signal. After the target task is revealed, safety violations are not allowed anymore. Thus, the guide is leveraged to compose a safe behaviour policy. Drawing from transfer learning, we also regularize a target policy (the student) towards the guide while the student is unreliable and gradually eliminate the influence of the guide as training progresses. The empirical analysis shows that this method can achieve safe transfer learning and helps the student solve the target task faster.

CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration

“…In the target tasks, the extrinsic environment reward is revealed to the agent. We leverage the safe exploration policy to guide learning in the off-policy safe guide (SaGui; Yang et al 2022a) framework, which achieves safe transfer learning by two mechanisms: i) Adaptively regularize the student policy to the guide policy based on the student's safety; ii) Use the safe exploration policy as a recovery policy when the student starts to take unsafe actions.…”

“…By maximizing the policy entropy, the agent trained by SAC-λ tends to have diverse behaviors, but it does not imply efficient exploration of the environment. With an additional intrinsic reward, the exploration of SAC-λ can be enhanced (Yang et al 2022a), but the interpretability of the learned policy in exploration is not clear. Achiam et al (2017); Liu, Ding, and Liu (2020); Yang et al (2020) propose a series of constrained policy optimization methods, where the constraints are built on longterm costs instead of real costs within a finite horizon.…”

“…The knowledge gained can make an agent easier to achieve a variety of tasks later. When employing a safe exploration policy as a safe guide (SaGui; Yang et al 2022a), an agent can adapt safely and quickly to a revealed task, especially when the task's reward signal is sparse. In this paper, we focus on learning such a task-agnostic safe exploration policy.…”

Energy-Efficient Cyber-Physical Systems: A Model-Based Approach

“…Using formal methods to analyse complex systems carefully has become a trend in robotics. For example, Yang et al suggested a technique called guided and safe reinforcement learning [66], and Pek et al use something called Spatio-Temporal Logic to plan and keep an eye on complex robot tasks [51]. Our work takes a different approach to planning how autonomous systems move.…”