Automating Control of Overestimation Bias for Reinforcement Learning

Abstract

Bias correction techniques are used by most of the high-performing methods for off-policy reinforcement learning. However, these techniques rely on a pre-defined bias correction policy that is either not flexible enough or requires environment-specific tuning of hyperparameters. In this work, we present a simple data-driven approach for guiding bias correction. We demonstrate its effectiveness on the Truncated Quantile Critics -a stateof-the-art continuous control algorithm. The proposed technique can adjust the bias correction across environments automatically. As a result, it eliminates the need for an extensive hyperparameter search, significantly reducing the actual number of interactions and computation.

Realistic Actor-Critic: A framework for balance between value overestimation and underestimation

“…Weighted bellman backup (Lee et al, 2021 ) and uncertainty-weighted actor-critic (Wu et al, 2021 ) prevent error propagation (Kumar et al, 2020 ) in Q-learning by reweighing sample transitions based on uncertainty estimations from the ensembles (Lee et al, 2021 ) or Monte Carlo dropout (Wu et al, 2021 ). AdaTQC (Kuznetsov et al, 2021 ) proposed an auto mechanism for controlling overestimation bias. Unlike prior works, our work does not reweight sample transitions but directly adds uncertainty estimations to punish the target value.…”

Learning Pessimism for Reinforcement Learning

“…More related to our method, Tactical Optimism and Pessimism (Moskovitz et al 2021) introduced the concept of adapting a bias penalty online. Together with similar later work (Kuznetsov et al 2021), they proposed step-wise updates to the bias correction parameters based on the performance of recent trajectories. Instead, GPL proposes a new method to precisely estimate bias and reduce its magnitude via dual gradient descent.…”