AdaPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems

Abstract

Model-free policy learning has enabled good performance on complex tasks that were previously intractable with traditional control techniques. However, this comes at the cost of requiring a perfectly accurate model for training. This is infeasible due to the very high sample complexity of model-free methods preventing training on the target system. This renders such methods unsuitable for physical systems. Model mismatch due to dynamics parameter differences and unmodeled dynamics error may cause suboptimal or unsafe behavior upon direct transfer. We introduce the Adaptive Policy Transfer for Stochastic Dynamics (ADAPT) algorithm that achieves provably safe and robust, dynamically-feasible zero-shot transfer of RL-policies to new domains with dynamics error. ADAPT combines the strengths of offline policy learning in a black-box source simulator with online tube-based MPC to attenuate bounded dynamics mismatch between the source and target dynamics. ADAPT allows online transfer of policies, trained solely in a simulation offline, to a family of unknown targets without fine-tuning. We also formally show that (i) ADAPT guarantees bounded state and control deviation through state-action tubes under relatively weak technical assumptions and, (ii) ADAPT results in a bounded loss of reward accumulation relative to a policy trained and evaluated in the source environment. We evaluate ADAPT on 2 continuous, non-holonomic simulated dynamical systems with 4 different disturbance models, and find that ADAPT performs between 50%-300% better on mean reward accrual than direct policy transfer.

BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learning

“…We use a five dimensional car that is a standard test environment in motion planning [35] and RL [5]. The car Fig.…”

Data-Efficient Multirobot, Multitask Transfer Learning for Trajectory Tracking

“…The multi-task transfer framework, discussed in the next subsection, requires a desired trajectory and the correct input that makes the system track the desired trajectory. To construct this pair of desired trajectory and corresponding correct input, we use an optimization-based ILC [21] to modify the input and improve the tracking performance of the system, which now behaves close to (8), in a small number of iterations 1, . .…”

“…where A L1 , B L1 and C L1 are the discrete-time matrices that describe (8). To capture the dynamics of a complete trial by a static mapping, we compute the lifted representation of (9) using Equations (13)(14) in [22] to obtain:…”

“…The transferred data improves learning of a model of a robotic arm by using data from a different robotic arm. In [8], an algorithm for robust transfer of learned policies to target environments with unmodeled disturbances or unknown parameters is developed. This method improved performance on unmodeled disturbances by 50-300%, compared to naive transfer.…”

Zero-shot Deep Reinforcement Learning Driving Policy Transfer for Autonomous Vehicles based on Robust Control

“…A method developed on a similar idea to the one in this paper is in [16], in which a MPC controller is designed to stabilize the target system around the nominal trajectory generated by consecutively applying the policy in the source system. Theorems on tube-based MPC ensure that the states are bounded under certain modeling error.…”