Transfer RL across Observation Feature Spaces via Model-Based Regularization

Abstract

In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features). However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefficient in terms of computation and sample complexity. Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer. Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task. Empirical results show that our algorithm significantly improves the efficiency and stability of learning in the target task. * The work was done while the author was an intern at Unity Technologies.

Provably Sample-Efficient RL with Side Information about Latent Dynamics

“…In transfer RL, most previous work [22,13] focuses on transferring under certain prior knowledge between the two observation spaces. Very recent work from Sun et al [21] is closest to the settings considered in this paper with a different solution than ours. Sun et al [21] tackled drastic changes in observation spaces and proposed an algorithm transferring the policy from source domain via learning a sufficient representation.…”

“…Very recent work from Sun et al [21] is closest to the settings considered in this paper with a different solution than ours. Sun et al [21] tackled drastic changes in observation spaces and proposed an algorithm transferring the policy from source domain via learning a sufficient representation. It provide an asymptotic guarantee of the policy learning given some representation condition, and empirical validation of the algorithm.…”

“…It provide an asymptotic guarantee of the policy learning given some representation condition, and empirical validation of the algorithm. However, Sun et al [21] does not gives finite-sample guarantees of the policy learning and any error bounds of learning the representation out of the deterministic transition case. Van Driessel and Francois-Lavet [24] proposed a deep RL algorithm for transfer learning with very different visual observation spaces by learning the abstract states, while this paper focuses more on a provable sample efficiency guarantee.…”

Iterative transfer learning for automatic collective motion tuning on multiple robot platforms

“…Since collective behavior data of simulated boids, simulated robots and real robots, have the same set of features, but with different observation space ranges and distributions (Abpeikar et al, 2022b ), the proposed method in this paper focuses on feature-based (observation space) transfer learning. Some feature-based transfer learning methods applied to RL are based on distribution similarity (Zhong et al, 2018 ), model-based regularization (Sun et al, 2022 ), and feature-space re-mapping (Feuz and Cook, 2015 ). The transfer learning on observation space used in this paper is based on using the Kullback-Leibler Divergence (KLD) method described by Zhong et al ( 2018 ).…”