Successive Over-Relaxation ${Q}$ -Learning

Abstract

In a discounted reward Markov Decision Process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is utilized to obtain the solution. In literature, a successive over-relaxation based value iteration scheme is proposed to speed-up the computation of the optimal value function. The speed-up is achieved by constructing a modified Bellman equation that ensures faster convergence to the optimal value function. However, in many practical applications, the model information is not known and we resort to Reinforcement Learning (RL) algorithms to obtain optimal policy and value function. One such popular algorithm is Q-learning. In this paper, we propose Successive Over-Relaxation (SOR) Q-learning. We first derive a modified fixed point iteration for SOR Q-values and utilize stochastic approximation to derive a learning algorithm to compute the optimal value function and an optimal policy. We then prove the almost sure convergence of the SOR Q-learning to SOR Q-values. Finally, through numerical experiments, we show that SOR Q-learning is faster compared to the standard Q-learning algorithm.

Generalized Speedy Q-Learning

“…The generalized Bellman operator can be used in other reinforcement learning algorithms as well. For example, it has already been applied to Watkins' Q-learning [10]. It will be interesting to study the rate of convergence and other properties of the modified algorithms, both theoretically and experimentally.…”

“…A popular method is successive overrelaxation (SOR). SOR technique has been applied previously to solve an MDP when the model information is completely known [5] and also in the setting of model-free reinforcement learning [10]. The latter algorithm is known as SOR Qlearning.…”

“…The latter algorithm is known as SOR Qlearning. The generalized Q-Bellman operator we use in this work is the same as the one used in [10].…”

“…It is proven, see [10], that H w is a max-norm contraction with contraction factor (1 − w + γw). That is, for w ∈ (0, w * ] and γ ∈ (0, 1), it is shown that 1 − w + γw ∈ [0, 1) and…”

“…It may be noted that the function Q * to which our algorithm converges could be different from the function Q ′ to which Watkins' Q-learning or Speedy Q-learning converges which is the fixed point of the Q-Bellman operator H defined by HQ(i, a) = r(i, a) + γ j∈S P (j|i, a) max b∈A Q(j, b) . However, it has been established in [10] that…”