Learning Safe Behaviour via Justified Human Preferences and Hypothetical Queries

Abstract

Although reinforcement learning is a powerful paradigm for agent sequential decision-making, it cannot be used in its traditional form in most safety-critical environments. Human feedback can enable an agent to learn a good policy while avoiding unsafe states, but at the cost of human time. We present JPAL-HA, a model for safe learning in safety-critical environments that is grounded on two novel ideas: (i) human preferences over a choice of actions are augmented with justifications such as one action is preferred because the other is unsafe; and (ii) we use these justifications to guide the generation of future queries over hypothetical actions to enable the agent to more effectively map out unsafe regions. Our algorithm comes with experimental safety and performance guarantees during training, and so offers a practical tool for users to train their agent safely and according to the level of safety their application requires. We evaluate it using a modified version of the Island Navigation environment from the AI Safety Gridworlds. We show that it significantly improves safety compared to existing human-in-the-loop safe reinforcement learning approaches and outperforms state-of-the-art learning algorithms in terms of sample efficiency. Further, we evaluate our method with people training a mobile robot, demonstrating that a diverse group of people can complete training in only a few minutes and that hypothetical actions not only improve safety, but also slightly decrease the training time and keep it to admissible levels.∗

A human-centered safe robot reinforcement learning framework with interactive behaviors

“…There are some prior works that investigate humancentered SRRL. For instance, Kazantzidis et al (2022) introduced a mechanism to ensure safety during exploration by harnessing human preferences. Reddy et al (2020) present a method to learn the model of human objectives by leveraging human feedback based on hypothetical behaviors, and then the model can be used to ensure the safety of robot learning.…”

“…The related problem of how to align has been discussed in earlier literature (Christiano et al, 2018;Leike et al, 2018;Kazantzidis et al, 2022;Liu R. et al, 2022) on how to align agents with user intentions, in which meaningful training signals can be hard to obtain, due to the unpredictable long-term effect of the behaviors, or potential influence to other agents and environments in large multi-agent systems.…”

“…Interaction is everywhere in human life (Kong et al, 2018), and agentenvironment interaction is the basis of RL (Sutton and Barto, 2018). When humans and robots act in a shared environment, their actions can be influenced by each other through interactive behaviors (Thomaz and Breazeal, 2006;Knox and Stone, 2009;MacGlashan et al, 2017;Kazantzidis et al, 2022;Lou et al, 2023). For example, a robot navigating in a public space can plan its path to avoid collisions with pedestrians and provide signals to pedestrians to move aside.…”

Cooperative Multi-Agent Learning in a Complex World: Challenges and Solutions

“…• Human-in-the-loop interactive learning for robust and trusted decision making, including studies on (Kazantzidis et al 2022;Gu et al 2022;Liu et al 2022).…”