A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

Suttle, Wesley; Yang, Zhuoran; Zhang, Kaiqing; Wang, Zhaoran; Başar, Tamer; Ji, Liu

doi:10.1016/j.ifacol.2020.12.2021

Cited by 35 publications

(40 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further work could be devoted to the weak convergence analysis of alternative multi-agent temporal difference schemes, including the emphatic temporal difference algorithm [38,44] and actor-critic algorithms [36,37]. Also, the proposed schemes could be extended to the cases of nonlinear value function approximations (such as using deep neural networks [27]).…”

Section: Discussionmentioning

confidence: 99%

“…The proposed distributed multi-agent algorithms can be considered as: 1) a tool for organizing coordinated actions of multiple agents contributing to the value function estimation and 2) a parallelization tool, allowing faster convergence, useful particularly in the problems with large dimensions. Notice that the in the first case the proposed algorithms can become a part of multi agent actor-critic schemes (see, e.g., [36,37]).…”

Section: Inter-agent Communications and Network Designmentioning

confidence: 99%

“…In [35] finite-time convergence rate of the algorithms is analyzed, while in [33] a modified algorithm with linear convergence rate is proposed. The authors of [32,36,37] developed multiagent distributed actor-critic schemes; the approach from [36] includes an iterative consensus-based emphatic temporal difference algorithm [38].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Distributed Value Function Approximation for Collaborative Multi-Agent Reinforcement Learning

Stanković¹,

Beko²,

Stanković³

2020

Preprint

View full text Add to dashboard Cite

In this paper we propose novel distributed gradient-based temporal difference algorithms for multi-agent off-policy learning of linear approximation of the value function in Markov decision processes. The algorithms are composed of: 1) local parameter updates based on the single-agent off-policy gradient temporal difference learning algorithms, including eligibility traces with state dependent parameters, and 2) linear dynamic consensus scheme over the underlying, typically sparsely connected, inter-agent communication network. The proposed algorithms differ in the way of how the time-scales are selected, how local recursions are performed and how consensus iterations are incorporated. The algorithms are completely decentralized, allowing applications in which all the agents may have completely different behavior policies while evaluating a single target policy. In this sense, the algorithms may be considered as a tool for either parallelization or multi-agent collaborative learning under given constraints. We provide weak convergence results, taking rigorously into account properties of the underlying Feller-Markov processes. We prove that, under nonrestrictive assumptions on the time-varying network topology and the individual state-visiting distributions of the agents, the parameter estimates of the algorithms weakly converge to a consensus point. The variance reduction effect of the proposed algorithms is demonstrated by analyzing a limiting stochastic differential equation. Specific guidelines for network design, providing the desired convergence points, are given. The algorithms' properties are illustrated by characteristic simulation results.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Inter-agent Communications and Network Designmentioning

confidence: 99%

See 1 more Smart Citation

Distributed Value Function Approximation for Collaborative Multi-Agent Reinforcement Learning

Stanković¹,

Beko²,

Stanković³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Finite-sample analyses on multi-agent reinforcement learning are also extended to fitted Q-iterations [Zhang et al, 2021]. For multi-agent actor-critic methods, the best available result is the proof of the asymptotic convergence [Suttle et al, 2020].…”

Section: Analysis Of Multi-agent Reinforcement Learningmentioning

confidence: 99%

Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation

Dong¹,

Shen²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study the convergence of the actor-critic algorithm with nonlinear function approximation under a nonconvex-nonconcave primal-dual formulation. Stochastic gradient descent ascent is applied with an adaptive proximal term for robust learning rates. We show the first efficient convergence result with primal-dual actor-critic with a convergence rate of O ln(N dG 2 ) Nunder Markovian sampling, where G is the element-wise maximum of the gradient, N is the number of iterations, and d is the dimension of the gradient. Our result is presented with only the Polyak-Lojasiewicz condition for the dual variables, which is easy to verify and applicable to a wide range of reinforcement learning (RL) scenarios. The algorithm and analysis are general enough to be applied to other RL settings, like multi-agent RL. Empirical results on OpenAI Gym continuous control tasks corroborate our theoretical findings.

show abstract

“…In the decentralized setting, a few works have obtained the almost sure convergence results of AC [17,21,22,35,36], but the finite-time convergence rate and complexity are still unexplored. To the best of our knowledge, there is no formally developed decentralized NAC algorithm.…”

Section: Related Workmentioning

confidence: 99%

Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis

Chen

Zhou

Chen

et al. 2021

Preprint

View full text Add to dashboard Cite

Actor-critic (AC) algorithms have been widely adopted in decentralized multi-agent systems to learn the optimal joint control policy. However, existing decentralized AC algorithms either do not preserve the privacy of agents or are not sample and communication-efficient. In this work, we develop two decentralized AC and natural AC (NAC) algorithms that are private, and sample and communicationefficient. In both algorithms, agents share noisy information to preserve privacy and adopt mini-batch updates to improve sample and communication efficiency. Particularly for decentralized NAC, we develop a decentralized Markovian SGD algorithm with an adaptive mini-batch size to efficiently compute the natural policy gradient. Under Markovian sampling and linear function approximation, we prove the proposed decentralized AC and NAC algorithms achieve the stateof-the-art sample complexities O( −2 ln −1 ) and O( −3 ln −1 ), respectively, and the same small communication complexity O( −1 ln −1 ). Numerical experiments demonstrate that the proposed algorithms achieve lower sample and communication complexities than the existing decentralized AC algorithm.Preprint. Under review.

show abstract

A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

Cited by 35 publications

References 10 publications

Distributed Value Function Approximation for Collaborative Multi-Agent Reinforcement Learning

Distributed Value Function Approximation for Collaborative Multi-Agent Reinforcement Learning

Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation

Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis

Contact Info

Product

Resources

About