Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation

Abstract

Despite the recent success of deep reinforcement learning (RL), domain adaptation remains an open problem. Although the generalization ability of RL agents is critical for the real-world applicability of Deep RL, zero-shot policy transfer is still a challenging problem since even minor visual changes could make the trained agent completely fail in the new task. To address this issue, we propose a two-stage RL agent that first learns a latent unified state representation (LUSR) which is consistent across multiple domains in the first stage, and then do RL training in one source domain based on LUSR in the second stage. The cross-domain consistency of LUSR allows the policy acquired from the source domain to generalize to other target domains without extra training. We first demonstrate our approach in variants of CarRacing games with customized manipulations, and then verify it in CARLA, an autonomous driving simulator with more complex and realistic visual observations. Our results show that this approach can achieve state-of-the-art domain adaptation performance in related RL tasks and outperforms prior approaches based on latent-representation based RL and image-to-image translation.

Learning controllable elements oriented representations for reinforcement learning

“…Most domain adaptation approaches are designed to deal with the changes to the observation distribution. Current domain adaptation methods can be roughly divided into three categories: domain randomization (Tobin et al, 2017;Sadeghi & Levine, 2017;James et al, 2019), image-to-image translation (Gamrian & Goldberg, 2018a;Tzeng et al, 2015;You et al, 2017;Zhang et al, 2018), and adaptation via aligned representations (Xing et al, 2021;Higgins et al, 2017;.…”

“…Ideally, representations in this latent space can share consistent semantic meanings no matter which domain they come from. For example, (Xing et al, 2021) explicitly splits the latent representations into domain-specific and domain-general features and then builds policy on the domain-general features to ignore domain-specific variations.…”

“…We compare OPA with other approaches designed for domain adaptation, including DARLA (Higgins et al, 2017), LUSR (Xing et al, 2021), UNIT4RL (Gamrian & Goldberg, 2018b) and LTMBR (Sun et al, 2022). DARLA relies on learning disentangled representations to achieve transfer.…”

“…For example, (Gamrian & Goldberg, 2018a;Tzeng et al, 2015;You et al, 2017) learn an image-to-image translation model that can map the observations from the target domain back into the source domain, and therefore the policy trained in the source domain is directly applicable when equipped with such translation. Some works (Xing et al, 2021;Higgins et al, 2017;) also learn such a mapping indirectly, in which the observations from the source and target domain are mapped into aligned representations.…”

Solving large-scale multi-agent tasks via transfer learning with dynamic state representation

“…TL has played an important role in accelerating singleagent RL by adapting learnt knowledge from past relevant tasks. 10,16,17 Inspired by this scenario, TL in MARL [18][19][20][21] is also studied with respect to transferring knowledge across multi-agent tasks to help improve the learning performance. The above work has two main directions: knowledge transfer across tasks and transfer among agents.…”