Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers

Abstract

We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. Our approach stems from the idea that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function. This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish source-domain transitions from target-domain transitions. Intuitively, the modified reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain. Said another way, the agent is penalized for transitions that would indicate that the agent is interacting with the source domain, rather than the target domain. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics. On discrete and continuous control tasks, we illustrate the mechanics of our approach and demonstrate its scalability to high-dimensional tasks. * Equal contribution.Preprint. Under review.

Solving the Real Robot Challenge using Deep Reinforcement Learning

“…It is likely that some fine-tuning of the policy on real data would greatly increase its robustness in the real environment, and developing a technique which could do so efficiently is one direction for future work. Similarly, domain adaptation techniques could be employed to produce a policy more capable of adapting to the real environment [8,9]. However, ideally the policy could be learned from scratch on the real system; a suitable simulator may not always be available.…”

Transfer RL across Observation Feature Spaces via Model-Based Regularization

“…But the problem studied in Zhang et al (2020a) is a multi-task setting where the agent aims to learn generalizable abstract states from a series of tasks. Another related topic is domain adaptation in RL (Higgins et al, 2017;Eysenbach et al, 2020;Zhang et al, 2020b), where the target observation space (e.g. real world) is different from the source observation (e.g.…”

“…changed dimension). Moreover, the aim of domain adaptation is usually zero-shot generalization to new observations, thus prior knowledge or a few samples of the target domain is often needed (Eysenbach et al, 2020).…”

Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain

“…By training two classifiers to capture the domain difference between the source domain and the target domain, Domain Adaptation with Rewards from Classifiers (DARC) [26] can train a near-optimal policies for the target domain on source domain. We get inspiration from it, using the same method to capture domain differences, and introducing them into imitation learning to solve our problems.…”

“…In practice, direct calculation of DD(s t , a t , s t+1 ) is technically unfeasible. We draw on [26]'s methodology which calculates the domain shift, since the domain shift quantification in [26] bears resemblance with DD(s t , a t , s t+1 ) in form.…”