A Convergent Off-Policy Temporal Difference Algorithm

Abstract

Learning the value function of a given policy (target policy) from the data samples obtained from a different policy (behavior policy) is an important problem in Reinforcement Learning (RL). This problem is studied under the setting of offpolicy prediction. Temporal Difference (TD) learning algorithms are a popular class of algorithms for solving the prediction problem. TD algorithms with linear function approximation are shown to be convergent when the samples are generated from the target policy (known as on-policy prediction). However, it has been well established in the literature that off-policy TD algorithms under linear function approximation diverge. In this work, we propose a convergent on-line off-policy TD algorithm under linear function approximation. The main idea is to penalize the updates of the algorithm in a way as to ensure convergence of the iterates. We provide a convergence analysis of our algorithm. Through numerical evaluations, we further demonstrate the effectiveness of our algorithm.

Regularized Q-learning

“…where Θ is the set of all deterministic policies, and ⊗ is the Kronecker product. Theorem A.2, given in Appendix A.2 of the supplemental material, is similar to Theorem 2 in Diddigi et al, 2019, and ensures such a property. Now, we can use Theorem 2.8 to establish stability of the overall system.…”

“…To address this problem, Diddigi et al, 2019 adds a regularization term to TD-learning in order to make it convergent. Since Q-learning can be interpreted as an off-policy TD-learning, we add a regularization term to Q-learning update motivated by Diddigi et al, 2019. This modification leads to the proposed algorithm as follows:…”

“…Note that letting η = 0, the above update is reduced to the standard Q-learning with linear function approximation in (3). The proposed algorithm is different from Diddigi et al, 2019 in the sense that a regularization term is applied to Q-learning instead of TD-learning.…”

“…Sutton et al, 2016 re-weights some states to match on-policy distribution to stabilize the off-policy TDlearning. Diddigi et al, 2019 uses l 2 regularization to propose a new convergent off-policy TD-learning algorithm.…”