Distributed Off-Policy Temporal Difference Learning Using Primal-Dual Method

Abstract

The goal of this paper is to provide theoretical analysis and additional insights on a distributed temporal-difference (TD)-learning algorithm for the multi-agent Markov decision processes (MDPs) via saddle-point viewpoints. The (single-agent) TD-learning is a reinforcement learning (RL) algorithm for evaluating a given policy based on reward feedbacks. In multi-agent settings, multiple RL agents concurrently behave, and each agent receives its local rewards. The goal of each agent is to evaluate a given policy corresponding to the global reward, which is an average of the local rewards by sharing learning parameters through random network communications. In this paper, we propose a distributed TDlearning based on saddle-point frameworks, and provide rigorous analysis of finite-time convergence of the algorithm and its solution based on tools in optimization theory. The results in this paper provide general and unified perspectives of the distributed policy evaluation problem, and theoretically complement the previous works.

New Versions of Gradient Temporal-Difference Learning

“…The other assumptions are also met. Therefore, the PDGD of Problem 8, given in (12), is globally asymptotically stable, and converges to its unique equilibrium point (θ * , λ * ), where λ * is defined in (11). This completes the proof.…”

“…In particular, GTD2, proposed in [4], can be interpreted as a stochastic primal-dual gradient dynamics (PDGD) of a convex-concave saddlepoint problem, and hence, its convergence analysis can be approached from a different angle using optimization theory [5], [6]. These interpretations were subsequently applied to distributed RL problems in [8], [9], [10], and [11].…”

“…As mentioned before, saddle-point perspectives of GTDs and RLs were introduced in [5] and [6] based on the Lagrangian duality [5] and Fenchel duality [6]. These ideas were applied to distributed RL problems in [8], [9], [10], and [11]. Even though they and the proposed saddle-point framework lead to the same algorithm, the latter one is derived from a slightly different way based on a simple constrained convex optimization formulation, which are compatible with techniques in [13].…”