Protective Policy Transfer

Abstract

Being able to transfer existing skills to new situations is a key capability when training robots to operate in unpredictable real-world environments. A successful transfer algorithm should not only minimize the number of samples that the robot needs to collect in the new environment, but also prevent the robot from damaging itself or the surrounding environment during the transfer process. In this work, we introduce a policy transfer algorithm for adapting robot motor skills to novel scenarios while minimizing serious failures. Our algorithm trains two control policies in the training environment: a task policy that is optimized to complete the task of interest, and a protective policy that is dedicated to keep the robot from unsafe events (e.g. falling to the ground). To decide which policy to use during execution, we learn a safety estimator model in the training environment that estimates a continuous safety level of the robot. When used with a set of thresholds, the safety estimator becomes a classifier for switching between the protective policy and the task policy. We evaluate our approach on four simulated robot locomotion problems and a 2D navigation problem and show that our method can achieve successful transfer to notably different environments while taking the robot's safety into consideration.

Safe Reinforcement Learning for Legged Locomotion

“…We formulate the problem of safe locomotion learning in the context of safe RL. Inspired by prior work [5], [6], [7], Catwalk Two-leg balance Fig. 1: We evaluate our algorithm in legged locomotion tasks: catwalk and two-leg balance.…”

“…Different from prior methods that learn a safety critic function which predicts the possibility of safe violations [5], [6], [7], we propose a model-based approach to determine when to switch between the two policies based on the knowledge about the system dynamics. In reality, it is often the case that the designer has some knowledge of the system dynamics at hand.…”