Offline Reinforcement Learning as Anti-exploration

Abstract

Offline Reinforcement Learning (RL) aims at learning an optimal control from a fixed dataset, without interactions with the system. An agent in this setting should avoid selecting actions whose consequences cannot be predicted from the data. This is the converse of exploration in RL, which favors such actions. We thus take inspiration from the literature on bonus-based exploration to design a new offline RL agent. The core idea is to subtract a prediction-based exploration bonus from the reward, instead of adding it for exploration. This allows the policy to stay close to the support of the dataset and practically extends some previous pessimism-based offline RL methods to a deep learning setting with arbitrary bonuses. We also connect this approach to a more common regularization of the learned policy towards the data. Instantiated with a bonus based on the prediction error of a variational autoencoder, we show that our simple agent is competitive with the state of the art on a set of continuous control locomotion and manipulation tasks.

Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL

“…We describe a theorem that answers the first question: soft policy iteration [17] with a penalized value function Q is equivalent to policy iteration regularized by KL π(s) π p (s) where π p (a|s) = softmax − p(s, a) . This theorem is a generalized version of the theorem shown in [39], which does not require unnecessary assumptions on the penalty function.…”

“…One straightforward solution to the over-estimation problem is directly penalizing Q estimation [39,6] with a penalty function p(s, a):…”

“…Therefore, we want to design a penalty function that resembles the oracle error. Since the estimation error is likely to occur more often for out-ofdistribution state-action pairs (s, a ) / ∈ D, a few ad-hoc methods have been proposed for the purpose of measuring the epistemic uncertainty of Q θ ; the aleatoric uncertainty of the transition dynamics model is suggested as a proxy for the epistemic uncertainty of Q θ [52], and generative models [39] or pseudometrics [6] are proposed with the purpose of distinguishing whether a particular (s, a) is in-distribution or not. However, it has not been thoroughly investigated how penalty functions affect the policy iteration process, nor what penalty functions are best for offline RL.…”

“…Another direction of research tackles offline RL problems via value learning, trying to resolve the distribution shift problem that arises in the offline setup. Since distribution shift commonly results in overestimation of values, the algorithms belonging to this category try to estimate values pessimistically for out-of-distribution inputs [29,14], sometimes by explicitly quantifying the certainty with a trained transition dynamics model [52,21] or a generative model [39,6]. Intuitively, value-based offline RL methods are preferred over the behavior cloning-based counterpart since the value function could tell both what should and should not be done while imitation learning only utilizes the half side of the information.…”

“…However, their use of MMD distance for the constraint is only empirically justified with certain conditions, such as the small number of samples in computing the distance [25]. Another common problem that arises in other penalty functions is their use of proxy and their formulation; for example, a conditional variational autoencoder (CVAE) [39], a pseudometric [6], or a transition dynamics model [52] are estimated instead of the behavior policy β, and penalty functions are designed heuristically with the proxy estimates. While such formulations could show some positive correlation to the suggested penalty function, there is no clear connection that allows us to interpret the penalty in terms of β or the support set.…”