Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization

Abstract

Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure. This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics. In this paper, we challenge this conditional independence assumption and propose a family of expressive autoregressive dynamics models that generate different dimensions of the next state and reward sequentially conditioned on previous dimensions. We demonstrate that autoregressive dynamics models indeed outperform standard feedforward models in log-likelihood on heldout transitions. Furthermore, we compare different model-based and model-free off-policy evaluation (OPE) methods on RL Unplugged, a suite of offline MuJoCo datasets, and find that autoregressive dynamics models consistently outperform all baselines, achieving a new state-of-the-art. Finally, we show that autoregressive dynamics models are useful for offline policy optimization by serving as a way to enrich the replay buffer through data augmentation and improving performance using model-based planning. * Work done as an intern at Google Brain.

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning

“…This form of evaluation is closer to a worst-case analysis than the typical average performance reporting. In this case, the conservative estimation is important, since there exists no equivalent to early stopping from supervised learning in the offline RL setting: Offline policy evaluation and selection are still open problems (Hans et al, 2011;Paine et al, 2020;Konyushkova et al, 2021;Zhang et al, 2021;Fu et al, 2021), so we need to stop the policy training at some random point and deploy that policy on the real system. The above procedure is meant to simulate this random stopping, and should quantify how good the algorithm performs at least.…”

Get a Head Start: On-Demand Pedagogical Policy Selection in Intelligent Tutoring

“…A variety of contemporary OPE methods has been proposed, which can be mainly divided into three categories ): (i) Inverse propensity scoring (Precup 2000), such as Importance Sampling (IS) (Doroudi, Thomas, and Brunskill 2017). (ii) Direct methods that directly estimate the value functions of the evaluation policy (Nachum et al 2019;Zhang et al 2021;Yang et al 2022), including but not limited to model-based estimators (MB) (Paduraru 2013;Zhang et al 2021), valuebased estimators (Le, Voloshin, and Yue 2019) such as Fitted Q Evaluation (FQE), and minimax estimators (Zhang, Liu, and Whiteson 2020; Yue 2021) such as DualDICE (Yang et al 2020a). (iii) Hybrid methods combine aspects of both inverse propensity scoring and direct methods (Thomas and Brunskill 2016), such as DR (Jiang and Li 2016).…”

User-Interactive Offline Reinforcement Learning

“…Offline policy evaluation or offline hyperparameter selection is a field concerned with mitigating the problem that offline RL algorithms cannot evaluate their policy on the true environment and thus risk deploying policies that perform worse than before: Essentially, these methods try to evaluate (or at least rank) policies that have been found by an offline RL algorithm, in order to pick the best performing one for deployment and / or to tune hyperparameters. Often, dynamics models are used to evaluate policies found in model-free algorithms, however also model-free evaluation methods exist (Hans et al, 2011;Paine et al, 2020;Konyushova et al, 2021;Zhang et al, 2021;Fu et al, 2021). Unfortunately, but also intuitively, this problem is rather hard since if any method is found that can accurately assess the policy performance (i.e.…”

“…Instead, we could do offline RL and derive a policy which potentially outperforms the users. However, in offline RL we cannot simply evaluate policy candidates on the real system and offline policy evaluation is still an open problem (Hans et al, 2011;Paine et al, 2020;Konyushova et al, 2021;Zhang et al, 2021;Fu et al, 2021). We thus run the risk of choosing the wrong proximity hyperparameter λ and achieving either lower rewards or not meeting the users personal preferences.…”