Fictitious Play in Zero-Sum Stochastic Games

Sayin, Muhammed O.; Parise, Francesca; Ozdaglar, Asuman

doi:10.1137/21m1426675

Cited by 19 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Computing and learning equilibria in Markov games has attracted considerable interest recently. Most focus has been on the Nash equilibrium in either identical-interestor more generally, potential-games (Fox et al 2022;Leonardos et al 2022;Aydin and Eksin 2023;Ding et al 2022;Zhang et al 2022b), or two-player zero-sum Markov games (Daskalakis, Foster, and Golowich 2020;Cen et al 2023;Wei et al 2021;Zhang et al 2020;Sayin et al 2021;Huang et al 2022;Cui and Du 2022;Perolat et al 2015;Zeng, Doan, and Romberg 2022;Pattathil, Zhang, and Ozdaglar 2023;Yang and Ma 2023), albeit with a few exceptions (Qin and Etesami 2023;Sayin 2023;Giannou et al 2022;Kalogiannis and Panageas 2023;Kalogiannis et al 2023;Park, Zhang, and Ozdaglar 2023). In general-sum multi-player games, in light of the intractability of Nash equilibria, most focus has been on computing or indeed learning (coarse) correlated equilibria (Daskalakis, Golowich, and Zhang 2023;Jin et al 2021;Erez et al 2023;Liu, Szepesvári, and Jin 2022;Zhang et al 2022a).…”

Section: Further Related Workmentioning

confidence: 99%

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence beyond the Minty Property

Anagnostides,

Panageas,

Farina

et al. 2024

AAAI

View full text Add to dashboard Cite

Policy gradient methods enjoy strong practical performance in numerous tasks in reinforcement learning. Their theoretical understanding in multiagent settings, however, remains limited, especially beyond two-player competitive and potential Markov games. In this paper, we develop a new framework to characterize optimistic policy gradient methods in multi-player Markov games with a single controller. Specifically, under the further assumption that the game exhibits an equilibrium collapse, in that the marginals of coarse correlated equilibria (CCE) induce Nash equilibria (NE), we show convergence to stationary epsilon-NE in O(1/epsilon^2) iterations, where O suppresses polynomial factors in the natural parameters of the game. Such an equilibrium collapse is well-known to manifest itself in two-player zero-sum Markov games, but also occurs even in a class of multi-player Markov games with separable interactions, as established by recent work. As a result, we bypass known complexity barriers for computing stationary NE when either of our assumptions fails. Our approach relies on a natural generalization of the classical Minty property that we introduce, which we anticipate to have further applications beyond Markov games.

show abstract

Section: Further Related Workmentioning

confidence: 99%

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence beyond the Minty Property

Anagnostides,

Panageas,

Farina

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…See Section 3.1 for more details. The error induced from such non-zero-sum structure appears in existing work Sayin et al (2021Sayin et al ( , 2022a, and was handled by designing a novel truncated Lyapunov function. However, the truncated Lyapunov function was sufficient to establish the asymptotic convergence, but did not provide the explicit rate at which the induced error goes to zero.…”

Section: Challenges and Our Techniquesmentioning

confidence: 99%

“…Best-response type independent learning for stochastic games has attracted increasing attention lately (Leslie et al, 2020;Sayin et al, 2021Sayin et al, , 2022aBaudin and Laraki, 2022b,a;Maheshwari et al, 2022), with Sayin et al (2021Sayin et al ( , 2022a; Baudin and Laraki (2022b,a) tackling the zero-sum setting. However, only asymptotic convergence was established in these works.…”

Section: Sample-efficient Marlmentioning

confidence: 99%

A Finite-Sample Analysis of Payoff-Based Independent Learning in Zero-Sum Stochastic Games

Chen¹,

Zhang²,

Mazumdar³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study two-player zero-sum stochastic games, and propose a form of independent learning dynamics called Doubly Smoothed Best-Response dynamics, which integrates a discrete and doubly smoothed variant of the best-response dynamics into temporal-difference (TD)-learning and minimax value iteration. The resulting dynamics are payoff-based, convergent, rational, and symmetric among players. Our main results provide finitesample guarantees. In particular, we prove the first-known Õ(1/ǫ 2 ) sample complexity bound for payoff-based independent learning dynamics, up to a smoothing bias. In the special case where the stochastic game has only one state (i.e., matrix games), we provide a sharper Õ(1/ǫ) sample complexity. Our analysis uses a novel coupled Lyapunov drift approach to capture the evolution of multiple sets of coupled and stochastic iterates, which might be of independent interest.

show abstract

“…Assuming a blockchain system with high throughput and high capacity, this paper proposes a future vision for a decentralized game (degame) in which gamefi is transformed into degame by weakening its financial nature and highlighting its gameplay [10].…”

Section: A Conception For a De-game With Achievable Performancementioning

confidence: 99%

Problem analysis and future vision for GameFi: A transition from GameFi to DeFi gaming

Pei

2023

ACE

View full text Add to dashboard Cite

The increasing popularity of GameFi, a combination of gaming and financial models, has led to the emergence of certain problems with current financial models that can only be overcome through a transition towards de-game, or full-on-chain games. This paper provides an overview of the history and financial models of GameFi, identifies pain points on the path of transition from GameFi to de-game, such as capacity and TPS, and presents a concept for a de-game with achievable performance levels. Both single and double currency models have limitations that prevent them from reaching their full potential, demonstrating the need for a transition to de-game, capable of offering unique gaming experiences and financial rewards. Overcoming pain points such as capacity and TPS will be essential in achieving this transition. Proposed solutions include using sharding technology and optimizing smart contract architecture to enhance system capacity and TPS. Ultimately, this paper highlights both the challenges and opportunities presented by GameFi, proposing the transition to de-game to overcome the limitations of current financial models. Addressing critical pain points such as capacity and TPS, our proposed solutions offer feasible approaches to building reliable on-chain gaming systems. The future vision of GameFi in the form of de-game offers a promising outlook for the gaming industry, where players can engage in immersive games with financial incentives.

show abstract

Fictitious Play in Zero-Sum Stochastic Games

Cited by 19 publications

References 30 publications

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence beyond the Minty Property

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence beyond the Minty Property

A Finite-Sample Analysis of Payoff-Based Independent Learning in Zero-Sum Stochastic Games

Problem analysis and future vision for GameFi: A transition from GameFi to DeFi gaming

Contact Info

Product

Resources

About