Gradient play in stochastic games: stationary points, convergence, and sample complexity

Zhang, Runyu; Ren, Zhaolin; Li, Na

doi:10.48550/arxiv.2106.00198

Cited by 4 publications

(21 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• We propose an independent policy gradient algorithm -Algorithm 1 -for learning an -Nash equilibrium of MPGs with O(1/ 2 ) iteration complexity. In contrast to the existing results (Leonardos et al, 2021;Zhang et al, 2021b), such iteration complexity does not explicitly depend on the state space size.…”

Section: Introductionmentioning

confidence: 60%

“…In this paper, we provide the first affirmative answer to this question for a class of mixed cooperative/competitive Markov games -the so-called Markov potential games (MPGs) (Macua et al, 2018;Leonardos et al, 2021;Zhang et al, 2021b). In particular, we make the following contributions:…”

Section: Introductionmentioning

confidence: 94%

“…MPGs have also attracted attention in multi-agent RL. In the infinite-horizon setting, Leonardos et al (2021); Zhang et al (2021b) extended the policy gradient method (Agarwal et al, 2021;Kakade, 2001) for multiple players and established the iteration/sample complexity that scales with the size of state space; Fox et al (2021) generalized the natural policy gradient method (Kakade, 2001;Agarwal et al, 2021) and established the global asymptotic convergence. In the finite-horizon setting, Song et al build on the single-agent Nash-VI (Liu et al, 2021) to propose a sample efficient turn-based algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…More importantly, independent learning dynamics do not scale exponentially with the number of players in the game. Recently, Daskalakis et al (2020); Leonardos et al (2021); Zhang et al (2021b) have in fact shown that multi-agent RL players could perform policy gradient updates independently, while enjoying global non-asymptotic convergence. However, these results are only focused on the basic tabular setting in which the value functions are represented by tables; they do not carry over to large-scale multi-agent RL problems in which the state space size is potentially infinite and the number of players is large.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Ding¹,

Wei²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We examine global non-asymptotic convergence properties of policy gradient methods for multiagent reinforcement learning (RL) problems in Markov potential games (MPG). To learn a Nash equilibrium of an MPG in which the size of state space and/or the number of players can be very large, we propose new independent policy gradient algorithms that are run by all players in tandem. When there is no uncertainty in the gradient evaluation, we show that our algorithm finds an -Nash equilibrium with O(1/ 2 ) iteration complexity which does not explicitly depend on the state space size. When the exact gradient is not available, we establish O(1/ 5 ) sample complexity bound in a potentially infinitely large state space for a sample-based algorithm that utilizes function approximation. Moreover, we identify a class of independent policy gradient algorithms that enjoys convergence for both zero-sum Markov games and Markov cooperative games with the players that are oblivious to the types of games being played. Finally, we provide computational experiments to corroborate the merits and the effectiveness of our theoretical developments.

show abstract

Section: Introductionmentioning

confidence: 60%

Section: Introductionmentioning

confidence: 94%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Ding¹,

Wei²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…An application of RL for finding NE in linear-quadratic mean-field games has been studied in [26]. Moreover [27], [28] show that n-player Markov potential games, an extension of static potential games to dynamic stochastic games, admit polynomial-time algorithms for computing their NE policies. Unfortunately, the class of Markov potential games is very restrictive because it requires strong assumptions on the existence of a general potential function.…”

mentioning

confidence: 99%

Learning Stationary Nash Equilibrium Policies in $n$-Player Stochastic Games with Independent Chains

Etesami¹

2022

Preprint

View full text Add to dashboard Cite

We consider a subclass of n-player stochastic games, in which players have their own internal state/action spaces while they are coupled through their payoff functions. It is assumed that players' internal chains are driven by independent transition probabilities. Moreover, players can only receive realizations of their payoffs but not the actual functions, nor can they observe each others' states/actions. Under some assumptions on the structure of the payoff functions, we develop efficient learning algorithms based on Dual Averaging and Dual Mirror Descent, which provably converge almost surely or in expectation to the set of ǫ-Nash equilibrium policies. In particular, we derive upper bounds on the number of iterates that scale polynomially in terms of the game parameters to achieve an ǫ-Nash equilibrium policy. Besides Markov potential games and linear-quadratic stochastic games, this work provides another interesting subclass of n-player stochastic games that provably admit polynomial-time learning algorithm for finding their ǫ-Nash equilibrium policies. Index TermsStochastic games; stationary Nash equilibrium, dual averaging, dual mirror descent, learning in games. I. INTRODUCTIONSince the early work on the existence of a mixed-strategy Nash equilibrium in static noncooperative games [1], and its extension on the existence of stationary Nash equilibrium policies in dynamic stochastic games [2], substantial research has been done to develop scalable algorithms for computing Nash equilibrium (NE) points in static and dynamic environments. NE provides a stable solution concept for strategic multiagent decision-making systems, which is a desirable property in many applications such as socioeconomic systems [3], network security [4], routing and scheduling [5], among many others [6], [7].Unfortunately, computing NE is generally PPAD-hard [8], and it is unlikely to admit a polynomial-time algorithm. Thus, to overcome this fundamental barrier, two main approaches have been adapted in the past literature: i) searching for relaxed notions of stable solutions such as correlated equilibrium [9], which includes the set of NE, and ii) searching for NE points in special structured games such as potential games [10], or concave games [11]. Thanks to recent advances in the field of learning theory, it is known that some tailored algorithms for finding relaxed notions of equilibrium in case (i) can also be used to compute NE points of structured games in case (ii). For instance, it is known that the so-called no-regret algorithms always converge to the set of coarse correlated equilibria [7], and they can also be used to compute NE in the class of socially concave games [12]. However, such results are mainly developed for static games, in which players repeatedly play the same game and gradually learn the underlying stationary environment. Unfortunately, extensions of such results to dynamic stochastic games [2], [13], in which the state of the game evolves as a result of players' past decisions and the realizations of a stoc...

show abstract

Fictitious Play in Markov Games with Single Controller

Sayin¹,

Zhang²,

Ozdaglar³

2022

Preprint

View full text Add to dashboard Cite

Certain but important classes of strategic-form games, including zero-sum and identical-interest games, have the fictitious-play-property (FPP), i.e., beliefs formed in fictitious play dynamics always converge to a Nash equilibrium (NE) in the repeated play of these games. Such convergence results are seen as a (behavioral) justification for the game-theoretical equilibrium analysis. Markov games (MGs), also known as stochastic games, generalize the repeated play of strategicform games to dynamic multi-state settings with Markovian state transitions. In particular, MGs are standard models for multi-agent reinforcement learning -a reviving research area in learning and games, and their game-theoretical equilibrium analyses have also been conducted extensively. However, whether certain classes of MGs have the FPP or not (i.e., whether there is a behavioral justification for equilibrium analysis or not) remains largely elusive. In this paper, we study a new variant of fictitious play dynamics for MGs and show its convergence to an NE in n-player identical-interest MGs in which a single player controls the state transitions. Such games are of interest in communications, control, and economics applications. Our result together with the recent results in [Sayin et al., 2020] establishes the FPP of two-player zero-sum MGs and n-player identical-interest MGs with a single controller (standing at two different ends of the MG spectrum from fully competitive to fully cooperative).

show abstract

Gradient play in stochastic games: stationary points, convergence, and sample complexity

Cited by 4 publications

References 40 publications

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Learning Stationary Nash Equilibrium Policies in $n$-Player Stochastic Games with Independent Chains

Fictitious Play in Markov Games with Single Controller

Contact Info

Product

Resources

About