Finite-Sample Analysis of Multi-Agent Policy Evaluation with Kernelized Gradient Temporal Difference

Heredia, Paulo; Mou, Shaoshuai

doi:10.1109/cdc42340.2020.9303966

Cited by 8 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Distributed TD-learning algorithms are studied in [8]- [14]. The results in [8], [9], [34] consider central rewards with different assumptions.…”

Section: B: Policy Evaluation Problemmentioning

confidence: 99%

“…The work in [13] develops the so-called homotopy stochastic primaldual algorithm with local actions and adaptive learning rates. A distributed gradient TD-learning with value functions that lie in reproducing kernel Hilbert spaces is proposed in [14] with its finite-sample analysis.…”

Section: B: Policy Evaluation Problemmentioning

confidence: 99%

“…Recently, significant progresses have been made in distributed RL schemes [3]- [14]. These recent researches apply different approaches and assumptions, and cover different scenarios.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Distributed Off-Policy Temporal Difference Learning Using Primal-Dual Method

Lee

Kim

2022

IEEE Access

View full text Add to dashboard Cite

The goal of this paper is to provide theoretical analysis and additional insights on a distributed temporal-difference (TD)-learning algorithm for the multi-agent Markov decision processes (MDPs) via saddle-point viewpoints. The (single-agent) TD-learning is a reinforcement learning (RL) algorithm for evaluating a given policy based on reward feedbacks. In multi-agent settings, multiple RL agents concurrently behave, and each agent receives its local rewards. The goal of each agent is to evaluate a given policy corresponding to the global reward, which is an average of the local rewards by sharing learning parameters through random network communications. In this paper, we propose a distributed TDlearning based on saddle-point frameworks, and provide rigorous analysis of finite-time convergence of the algorithm and its solution based on tools in optimization theory. The results in this paper provide general and unified perspectives of the distributed policy evaluation problem, and theoretically complement the previous works.

show abstract

“…Distributed TD-learning algorithms are studied in [8]- [14]. The results in [8], [9], [34] consider central rewards with different assumptions.…”

Section: B: Policy Evaluation Problemmentioning

confidence: 99%

Section: B: Policy Evaluation Problemmentioning

confidence: 99%

See 1 more Smart Citation

Distributed Off-Policy Temporal Difference Learning Using Primal-Dual Method

Lee

Kim

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Similar bounds have also been derived in Chen et al [2021] for two distributed variants of the TDC method. There are also some other works that derive finite-time bounds Wai et al [2018], Ding et al [2019], Xu et al [2020], Zhao et al [2020], Heredia andMou [2020], Stanković et al [2020], Ren et al [2021], Zhang et al [2021b], but we do not discuss them in this paper since the algorithms proposed there do not fit the update rule given in (2).…”

Section: Related Workmentioning

confidence: 99%

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Thoppe¹,

Kumar²

2021

Preprint

View full text Add to dashboard Cite

In Multi-Agent Reinforcement Learning (MARL), multiple agents interact with a common environment, as also with each other, for solving a shared problem in sequential decision-making. It has wide-ranging applications in gaming, robotics, finance, etc. In this work, we derive a novel law of iterated logarithm for a family of distributed nonlinear stochastic approximation schemes that is useful in MARL. In particular, our result describes the convergence rate on almost every sample path where the algorithm converges. This result is the first of its kind in the distributed setup and provides deeper insights than the existing ones, which only discuss convergence rates in the expected or the CLT sense. Importantly, our result holds under significantly weaker assumptions: neither the gossip matrix needs to be doubly stochastic nor the stepsizes square summable. As an application, we show that, for the stepsize n −γ with γ ∈ (0, 1), the distributed TD(0) algorithm with linear function approximation has a convergence rate of O( √ n −γ ln n) a.s.; for the 1/n type stepsize, the same is O( √ n −1 ln ln n) a.s. These decay rates do not depend on the graph depicting the interactions among the different agents.Preprint. Under review.

show abstract

“…In the aforementioned references (limited to discounted objectives), convergence guarantees are mostly asymptotic, apply only to MARL sub-problems as policy evaluation (estimating the value function assuming a fixed policy (Heredia & Mou, 2020;Sha et al, 2020)), or due to implied non-convexity induced by policy parameterization, cannot avoid spurious 1 policies (Qu et al, 2020a,b) -see (Zhang et al, 2020) for further details.…”

Section: Introductionmentioning

confidence: 99%

Convergence Rates of Average-Reward Multi-agent Reinforcement Learning via Randomized Linear Programming

Koppel¹,

Bedi²,

Bhargav³

et al. 2021

Preprint

View full text Add to dashboard Cite

In tabular multi-agent reinforcement learning with average-cost criterion, a team of agents sequentially interacts with the environment and observes local incentives. We focus on the case that the global reward is a sum of local rewards, the joint policy factorizes into agents' marginals, and full state observability. To date, few global optimality guarantees exist even for this simple setting, as most results yield convergence to stationarity for parameterized policies in large/possibly continuous spaces. To solidify the foundations of MARL, we build upon linear programming (LP) reformulations, for which stochastic primal-dual methods yields a model-free approach to achieve optimal sample complexity in the centralized case. We develop multiagent extensions, whereby agents solve their local saddle point problems and then perform local weighted averaging. We establish that the sample complexity to obtain nearglobally optimal solutions matches tight dependencies on the cardinality of the state and action spaces, and exhibits classical scalings with respect to the network in accordance with multi-agent optimization. Experiments corroborate these results in practice. * denotes equal contributions.

show abstract

Finite-Sample Analysis of Multi-Agent Policy Evaluation with Kernelized Gradient Temporal Difference

Cited by 8 publications

References 22 publications

Distributed Off-Policy Temporal Difference Learning Using Primal-Dual Method

Distributed Off-Policy Temporal Difference Learning Using Primal-Dual Method

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Convergence Rates of Average-Reward Multi-agent Reinforcement Learning via Randomized Linear Programming

Contact Info

Product

Resources

About