Decentralized No-regret Learning Algorithms for Extensive-form Correlated Equilibria (Extended Abstract)

Abstract

The existence of uncoupled no-regret learning dynamics converging to correlated equilibria in normal-form games is a celebrated result in the theory of multi-agent systems. Specifically, it has been known for more than 20 years that when all players seek to minimize their internal regret in a repeated normal-form game, the empirical frequency of play converges to a normal-form correlated equilibrium. Extensive-form games generalize normal-form games by modeling both sequential and simultaneous moves, as well as imperfect information. Because of the sequential nature and the presence of private information, correlation in extensive-form games possesses significantly different properties than in normal-form games. The extensive-form correlated equilibrium (EFCE) is the natural extensive-form counterpart to the classical notion of correlated equilibrium in normal-form games. Compared to the latter, the constraints that define the set of EFCEs are significantly more complex, as the correlation device ({\em a.k.a.} mediator) must take into account the evolution of beliefs of each player as they make observations throughout the game. Due to this additional complexity, the existence of uncoupled learning dynamics leading to an EFCE has remained a challenging open research question for a long time. In this article, we settle that question by giving the first uncoupled no-regret dynamics which provably converge to the set of EFCEs in n-player general-sum extensive-form games with perfect recall. We show that each iterate can be computed in time polynomial in the size of the game tree, and that, when all players play repeatedly according to our learning dynamics, the empirical frequency of play after T game repetitions is guaranteed to be a O(T^-1/2)-approximate EFCE with high probability, and an EFCE almost surely in the limit.

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

“…Extensive-form games Finally, we remark that there is another long line of research on MARL based on the model of extensive-form games (EFG) [see, e.g., 28,19,65,9,10,12]. Results on learning EFGs do not directly imply results for learning MGs, since EFGs are naturally treestructured games, which can not efficiently represent MGs-graph-structured games where a state at the h th step can be the child of multiple states at the (h − 1) th step.…”

Simple Uncoupled No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium

“…For instance, we have that 135 = 145 , and 145 = 145 . We are now ready to define the concept of trigger regret minimization, which extends and generalizes the homonymous notion in the conference version of this paper [9], as well as the notion of internal regret minimization in normal-form games. Definition 3.6.…”

Learning Fair Equilibria in Sponsored Search Auctions

“…Moreover, given that no-external regret learning dynamics converge to the set of coarse correlated equilibria (see [8,20]), Blum et al [9] study the price of total anarchy in games in which players take decisions so as to minimize their external regret. No-regret learning dynamics converging to correlated equilibria have been studied in various settings (see, e.g., [36,42,18,19,32]). Moreover, Hartline et al [43] and Caragiannis et al [15] study the quality of outcomes emerging from no-regret dynamics in Bayesian settings.…”