Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage

Abstract

This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions. Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy. We introduce Model-based IL from Offline data (MILO): an algorithmic framework that utilizes the static dataset to solve the offline IL problem efficiently both in theory and in practice. In theory, even if the behavior policy is highly sub-optimal compared to the expert, we show that as long as the data from the behavior policy provides sufficient coverage on the expert state-action traces (and with no necessity for a global coverage over the entire state-action space), MILO can provably combat the covariate shift issue in IL. Complementing our theory results, we also demonstrate that a practical implementation of our approach mitigates covariate shift on benchmark MuJoCo continuous control tasks. We demonstrate that with behavior policies whose performances are less than half of that of the expert, MILO still successfully imitates with an extremely low number of expert state-action pairs while traditional offline IL method such as behavior cloning (BC) fails completely. Source code is provided at https://github.com/jdchang1/milo. * Equal contribution † Work done outside of Amazon 3 in this work, we use cost instead of reward, thus we call A π disadvantage function.

Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation

“…It means that, with the increasing d, we can enrich the reward set R and then capture the performance of the policy more meticulously using the optimality gap defined in (2.6). Analysis in GAIL with linear reward set is more challenging than the case with bounded reward set, which is studied in the previous literature (Shani et al, 2021;Chang et al, 2021).…”

“…Furthermore, previous theoretical analyses on GAIL either focus on the tabular case (Shani et al, 2021), where the state and action spaces are discrete, or relies on strong assumptions, including access to a wellexplored dataset (Zhang et al, 2020), linear-quadratic regulators (Cai et al, 2019), or kernelized nonlinear regulators (Chang et al, 2021). Theoretical analysis for GAIL with linear function approximation either in online or offline settings still remains an open problem, which is crucial for the application of GAIL in the continuous or high dimensional state and action spaces.…”

“…Related Works Our work adds to the body of analysis on GAIL (Cai et al, 2019;Chen et al, 2020;Zhang et al, 2020;Xu et al, 2020;Shani et al, 2021;Chang et al, 2021). Shani et al (2021) study online GAIL and obtain O(H 2 |S||A| 1/2 K 1/2 + H 3/2 |S| 1/2 |A| 1/2 KN −1/2 1 ) regret in the tabular case with bounded reward functions but we apply linear function approximation on the transition kernels without assuming the state space or the action space is discrete and we adopt linear reward set.…”

“…Cai et al (2019) study the global convergence of offline GAIL in the setting of linear-quadratic regulators, which is unnecessary for this paper. As an independent work, Chang et al (2021) study offline GAIL with bounded reward functions in the continuous kernelized nolinear regulator (KNR, Kakade et al (2020)) and Gaussian process (GP, Fisac et al (2018)) setting. We point out that the KNR (resp.…”