Lipschitz Lifelong Reinforcement Learning

Abstract

We consider the problem of knowledge transfer when an agent is facing a series of Reinforcement Learning (RL) tasks. We introduce a novel metric between Markov Decision Processes and establish that close MDPs have close optimal value functions. Formally, the optimal value functions are Lipschitz continuous with respect to the tasks space. These theoretical results lead us to a value transfer method for Lifelong RL, which we use to build a PAC-MDP algorithm with improved convergence rate. We illustrate the benefits of the method in Lifelong RL experiments.

Towards Safe Policy Improvement for Non-Stationary MDPs

“…To avoid arbitrary changes, previous works typically require the transition function P k and the reward function R k to be Lipschitz smooth over time [39,30,40,16]. And in fact, we can provide a bound on the change in performance given such Lipschitz conditions, as we show below in Theorem 1.…”