Meta Reinforcement Learning for Sim-to-real Domain Adaptation

Abstract

Modern reinforcement learning methods suffer from low sample efficiency and unsafe exploration, making it infeasible to train robotic policies entirely on real hardware. In this work, we propose to address the problem of sim-toreal domain transfer by using meta learning to train a policy that can adapt to a variety of dynamic conditions, and using a task-specific trajectory generation model to provide an action space that facilitates quick exploration. We evaluate the method by performing domain adaptation in simulation and analyzing the structure of the latent space during adaptation. We then deploy this policy on a KUKA LBR 4+ robot and evaluate its performance on a task of hitting a hockey puck to a target. Our method shows more consistent and stable domain adaptation than the baseline, resulting in better overall performance.

Few-Shot Model-Based Adaptation in Noisy Conditions

“…The comparison is performed both against gradient-based adaptation methods [5], [8] and against an LSTM baseline [14], which uses a blackbox adaptation scheme and does not explicitly account for noise.…”

“…Previous work on meta-learning for sim-to-real transfer focused on policy adaptation [5], [6], [22]. With these approaches, the final policy is a direct result of an on-policy update performed on the policy used to collect the data, as in [8].…”

“…We used 16 dimensions for the measurement and dynamics representations z and φ. Following prior work on using generative models and latent space trajectory representations [1], [5], [27], we use a variational autoencoder trained on a set of task-specific trajectories provided by human experts. This approach turns a sequential decision making problem into a multi-armed bandit problem and provides a straightforward way of collecting data for model adaptation-as the latent distribution is known to be a unit Gaussian, new trajectories can be generated by simply sampling latent vectors from that distribution and passing the latent values to the decoder.…”

“…For the sim-to-real experiments, we used the same hockey puck setup as in [5]. This setup consists of a KUKA LBR4+ robot arm equipped with a floorball stick, a horizontally placed whiteboard, and a two hockey pucks with different masses and friction characteristics: an inline hockey puck and an ice hockey puck.…”

“…Our goal is to enhance sim-to-real transfer of dynamic skills in terms of data-efficiency and speed of adaptation through uncertaintyawareness. In contrast to previously explored gradient-based methods [5], [6], [8], which merely look at the mean gradient direction (ignoring the variance information), we propose to use a memory-based approach, which allow the model to keep track of uncertainty statistics [10]. This uncertainty can be projected directly onto the predictions, as shown in Figure 1; before adaptation (Figure 1a) the model reports large uncertainty about the position of the hockeypuck after a planned hit, while after adaptation the uncertainty falls down (Figure 1b).…”