Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

Abstract

Popular off-policy deep reinforcement learning algorithms compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns. In this work, we propose a novel learnable penalty to enact such pessimism, based on a new way to quantify the critic's epistemic uncertainty. Furthermore, we propose to learn the penalty alongside the critic with dual TD-learning, a strategy to estimate and minimize the bias magnitude in the target returns. Our method enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets. Empirically, by integrating our method and other orthogonal improvements with popular off-policy algorithms, we achieve state-of-the-art results in continuous control tasks from both proprioceptive and pixel observations.

Learning Pessimism for Reinforcement Learning

“…Moreover, we implement the critic's ensemble as a single neural network, using linear non-fullyconnected layers evenly splitting the nodes and dropping the weight connections between the splits. Practically, when evaluated under the same hardware, this results in our algorithm running more than 2.4 times faster than the implementation from Chen et al (2021) while having a similar algorithmic complexity (see (Cetin and Celiktutan 2021)). We show that GPL significantly improves the performance and robustness of off-policy RL, concretely surpassing prior algorithms and setting new state-of-the-art results.…”

“…An intuition for our results is that the relative difference between the off-policy and on-policy action-value predictions should always push β to counteract new bias stemming from model errors in the policy gradient action maximization, and thus improve over non-adaptive methods which are also affected by initial bias. We further validate dual TD-learning in (Cetin and Celiktutan 2021), comparing with optimizing β by minimizing the squared norm of the bias and by using the bandit-based optimization from TOP (Moskovitz et al 2021). We also note that integrating GPL adds non-trivial complexity by introducing an entirely new optimization step which could be unnecessary for low-dimensional and easy-exploration problems.…”

“…Instead, GPL proposes a new method to precisely estimate bias and reduce its magnitude via dual gradient descent. We provide a direct empirical comparison in the extended version of this work (Cetin and Celiktutan 2021).…”

“…However, in practice, we still find that GPL's performance and the optimization dynamics are robust to different levels of initial target bias, and that dual TD-learning appears to always improve over fixed penalties. See (Cetin and Celiktutan 2021) for an analysis of the empirical behavior of GPL with different initial values of β and, thus, different bias in the initial targets. An intuition for our results is that the relative difference between the off-policy and on-policy action-value predictions should always push β to counteract new bias stemming from model errors in the policy gradient action maximization, and thus improve over non-adaptive methods which are also affected by initial bias.…”

Automating Control of Overestimation Bias for Reinforcement Learning

“…In contrast, our method relies directly on the bias estimation to adjust control hyperparameter. GPL-SAC [28] proposed a new overestimation control technique based on uncertainty estimates and adaptation technique simultaneously. Adaptation technique represents particular case (k = 1) of our approach.…”