Finite-Sample Analysis of Decentralized Q-Learning for Stochastic Games

Gao, Zuguang; Ma, Qiang; Başar, Tamer; Birge, John R.

doi:10.48550/arxiv.2112.07859

Cited by 3 publications

(3 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dynamic presented can converge to a (stationary pure-strategy) equilibrium if the associated normal-form game is weakly acyclic with respect to best (or better) response dynamics. The finite-sample complexity of the algorithm is also established recently in [Gao et al, 2021]. In contrast, our learning dynamic can converge to a stationary mixed-strategy equilibrium, which is essential for a global convergence result across the MG spectrum, as a pure-strategy equilibrium does not exist in general, e.g., in zero-sum games.…”

Section: Independent Learning In Mgsmentioning

confidence: 89%

Fictitious Play in Markov Games with Single Controller

Sayin¹,

Zhang²,

Ozdaglar³

2022

Preprint

View full text Add to dashboard Cite

Certain but important classes of strategic-form games, including zero-sum and identical-interest games, have the fictitious-play-property (FPP), i.e., beliefs formed in fictitious play dynamics always converge to a Nash equilibrium (NE) in the repeated play of these games. Such convergence results are seen as a (behavioral) justification for the game-theoretical equilibrium analysis. Markov games (MGs), also known as stochastic games, generalize the repeated play of strategicform games to dynamic multi-state settings with Markovian state transitions. In particular, MGs are standard models for multi-agent reinforcement learning -a reviving research area in learning and games, and their game-theoretical equilibrium analyses have also been conducted extensively. However, whether certain classes of MGs have the FPP or not (i.e., whether there is a behavioral justification for equilibrium analysis or not) remains largely elusive. In this paper, we study a new variant of fictitious play dynamics for MGs and show its convergence to an NE in n-player identical-interest MGs in which a single player controls the state transitions. Such games are of interest in communications, control, and economics applications. Our result together with the recent results in [Sayin et al., 2020] establishes the FPP of two-player zero-sum MGs and n-player identical-interest MGs with a single controller (standing at two different ends of the MG spectrum from fully competitive to fully cooperative).

show abstract

Section: Independent Learning In Mgsmentioning

confidence: 89%

Fictitious Play in Markov Games with Single Controller

Sayin¹,

Zhang²,

Ozdaglar³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Such asymmetric policy gradient methods are not completely independent, as some implicit coordination is required to enable such a timescale separation across agents. This style of implicit coordination is also required for the finite-sample analysis of decentralized learning in certain general-sum stochastic games, e.g., Gao et al (2021), which improves the asymptotic convergence in Arslan and Yüksel (2017).…”

Section: Sample-efficient Marlmentioning

confidence: 99%

A Finite-Sample Analysis of Payoff-Based Independent Learning in Zero-Sum Stochastic Games

Chen¹,

Zhang²,

Mazumdar³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study two-player zero-sum stochastic games, and propose a form of independent learning dynamics called Doubly Smoothed Best-Response dynamics, which integrates a discrete and doubly smoothed variant of the best-response dynamics into temporal-difference (TD)-learning and minimax value iteration. The resulting dynamics are payoff-based, convergent, rational, and symmetric among players. Our main results provide finitesample guarantees. In particular, we prove the first-known Õ(1/ǫ 2 ) sample complexity bound for payoff-based independent learning dynamics, up to a smoothing bias. In the special case where the stochastic game has only one state (i.e., matrix games), we provide a sharper Õ(1/ǫ) sample complexity. Our analysis uses a novel coupled Lyapunov drift approach to capture the evolution of multiple sets of coupled and stochastic iterates, which might be of independent interest.

show abstract

“…The asymptotic convergence of Q-learning with linear function approximation was established in [180] under a "negative drift" assumption. Under similar assumptions, the finite-sample analysis of Q-learning, as well as its on-policy variant SARSA, was performed in [49,185,183,186] for using linear function approximation, and in [187,118] for using neural network approximation. However, such negative drift assumption is highly artificial, highly restrictive, and is impossible to satisfy unless the discount factor of the MDP is extremely small (see Chapter 11).…”

Section: Related Literaturementioning

confidence: 99%

A Unified Lyapunov Framework for Finite-Sample Analysis of Reinforcement Learning Algorithms

Chen

2022

SIGMETRICS Perform. Eval. Rev.

View full text Add to dashboard Cite

Reinforcement learning (RL) is a paradigm where an agent learns to accomplish tasks by interacting with the environment, similar to how humans learn. RL is therefore viewed as a promising approach to achieve artificial intelligence, as evidenced by the remarkable empirical successes. However, many RL algorithms are theoretically not well-understood, especially in the setting where function approximation and off-policy sampling are employed. My thesis [1] aims at developing thorough theoretical understanding to the performance of various RL algorithms through finite-sample analysis. Since most of the RL algorithms are essentially stochastic approximation (SA) algorithms for solving variants of the Bellman equation, the first part of thesis is dedicated to the analysis of general SA involving a contraction operator, and under Markovian noise. We develop a Lyapunov approach where we construct a novel Lyapunov function called the generaled Moreau envelope. The results on SA enable us to establish finite-sample bounds of various RL algorithms in the tabular setting (cf. Part II of the thesis) and when using function approximation (cf. Part III of the thesis), which in turn provide theoretical insights to several important problems in the RL community, such as the efficiency of bootstrapping, the bias-variance trade-off in off-policy learning, and the stability of off-policy control. The main body of this document provides an overview of the contributions of my thesis.

show abstract

Finite-Sample Analysis of Decentralized Q-Learning for Stochastic Games

Cited by 3 publications

References 62 publications

Fictitious Play in Markov Games with Single Controller

Fictitious Play in Markov Games with Single Controller

A Finite-Sample Analysis of Payoff-Based Independent Learning in Zero-Sum Stochastic Games

A Unified Lyapunov Framework for Finite-Sample Analysis of Reinforcement Learning Algorithms

Contact Info

Product

Resources

About