A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Thoppe, Gugan; Kumar, Bhumesh

doi:10.1109/icc54714.2021.9702912

Cited by 3 publications

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Multi-Agent Natural Actor-Critic Reinforcement Learning Algorithms

Trivedi

Hemachandra

2022

Dyn Games Appl

View full text Add to dashboard Cite

Multi-agent actor-critic algorithms are an important part of the Reinforcement Learning (RL) paradigm. We propose three fully decentralized multi-agent natural actor-critic (MAN) algorithms in this work. The objective is to collectively find a joint policy that maximizes the average long-term return of these agents. In the absence of a central controller and to preserve privacy, agents communicate some information to their neighbors via a time-varying communication network. We prove convergence of all the three MAN algorithms to a globally asymptotically stable set of the ODE corresponding to actor update; these use linear function approximations. We show that the Kullback–Leibler divergence between policies of successive iterates is proportional to the objective function’s gradient. We observe that the minimum singular value of the Fisher information matrix is well within the reciprocal of the policy parameter dimension. Using this, we theoretically show that the optimal value of the deterministic variant of the MAN algorithm at each iterate dominates that of the standard gradient-based multi-agent actor-critic (MAAC) algorithm. To our knowledge, it is the first such result in multi-agent reinforcement learning (MARL). To illustrate the usefulness of our proposed algorithms, we implement them on a bi-lane traffic network to reduce the average network congestion. We observe an almost 25% reduction in the average congestion in 2 MAN algorithms; the average congestion in another MAN algorithm is on par with the MAAC algorithm. We also consider a generic 15 agent MARL; the performance of the MAN algorithms is again as good as the MAAC algorithm.

show abstract

Multi-Agent Natural Actor-Critic Reinforcement Learning Algorithms

Trivedi

Hemachandra

2022

Dyn Games Appl

View full text Add to dashboard Cite

show abstract

Online Learning with Adversaries: A Differential-Inclusion Analysis

Ganesh,

Reiffers-Masson,

Thoppe

2023

2023 62nd IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

We introduce an observation-matrix-based framework for fully asynchronous online Federated Learning (FL) with adversaries. In this work, we demonstrate its effectiveness in estimating the mean of a random vector. Our main result is that the proposed algorithm almost surely converges to the desired mean µ. This makes ours the first asynchronous FL method to have an a.s. convergence guarantee in the presence of adversaries. We derive this convergence using a novel differential-inclusion-based two-timescale analysis. Two other highlights of our proof include (a) the use of a novel Lyapunov function to show that µ is the unique global attractor for our algorithm's limiting dynamics, and (b) the use of martingale and stopping-time theory to show that our algorithm's iterates are almost surely bounded.

show abstract

Convergence of Momentum-based Distributed Stochastic Approximation with RL Applications

Naskar,

Thoppe

2023

2023 Ninth Indian Control Conference (ICC)

View full text Add to dashboard Cite

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Cited by 3 publications

References 6 publications

Multi-Agent Natural Actor-Critic Reinforcement Learning Algorithms

Multi-Agent Natural Actor-Critic Reinforcement Learning Algorithms

Online Learning with Adversaries: A Differential-Inclusion Analysis

Convergence of Momentum-based Distributed Stochastic Approximation with RL Applications

Contact Info

Product

Resources

About