Safe Deep RL in 3D Environments using Human Feedback

Abstract

Agents should avoid unsafe behaviour during both training and deployment. This typically requires a simulator and a procedural specification of unsafe behaviour. Unfortunately, a simulator is not always available, and procedurally specifying constraints can be difficult or impossible for many real-world tasks. A recently introduced technique, ReQueST, aims to solve this problem by learning a neural simulator of the environment from safe human trajectories, then using the learned simulator to efficiently learn a reward model from human feedback. However, it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans -whether sufficient pixel-based neural simulator quality can be achieved, and whether the human data requirements are viable in terms of both quantity and quality. In this paper we answer this question in the affirmative, using ReQueST to train an agent to perform a 3D first-person object collection task using data entirely from human contractors. We show that the resulting agent exhibits an order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning.

Learning Safe Behaviour via Justified Human Preferences and Hypothetical Queries

“…In other settings, we could assume that the agent (e.g. a robot) uses an attached camera to obtain the observations [14]. Finally, here we do not consider any debatable (normative) occasions, i.e.…”

“…We compare JPAL-HA as closely as possible with Human Intervention RL (HIRL) [24]. We did not compare with ReQueST [14,25], another human-inthe-loop method with a focus on safety, because the necessary approximations would make a comparison unrealistic; for example, ReQueST uses off-policy data and there is a question of how these data can be collected. HIRL [24] learns in three stages: (i) the agent learns a policy using standard reinforcement learning methods with a human overseeing it, blocking catastrophic actions and proposing an alternative (normally the best when they understand the environment) action; (ii) the agent pauses and a binary catastrophe classifier is built from the record of interventions from Stage (i) using supervised learning; and (iii) the agent starts again, but now the human is substituted by the catastrophe classifier (a kind of governor).…”

“…This is the work from which Hypothetical Actions were inspired for JPAL-HA, with the main difference being that we sample hypothetical actions from the policy model directly in critical situations. The ReQueST framework was scaled up in [14] to complex 3D safety-critical environments using, instead of reward shaping, a more sample-efficient type of feedback, reward sketching [35]. In the latter, humans are shown a video sequence of frames and are asked to draw a 'reward curve' based on the progress of the task.…”

“…Thus, one potential avenue for future research is to use the ReQueST framework (Section 6.1), (i.e. learning a dynamics model from expert safe trajectories, sampling trajectories from this model to generate queries for a human to provide feedback, and repeatedly improving the reward function before rendering it for traditional RL) but, instead of using Reward Shaping [25] or Reward Sketches [14] as the type of human feedback, to use human preferences. In that case, it would be able to exploit the safety benefits of justifications and hypothetical actions.…”