Generalized Bandit Regret Minimizer Framework in Imperfect Information Extensive-Form Game

Abstract

Regret minimization methods are a powerful tool for learning approximate Nash equilibrium (NE) in two-player zero-sum imperfect information extensive-form games (IIEGs). We consider the problem in the interactive bandit-feedback setting where we don't know the dynamics of the IIEG. In general, only the interactive trajectory and the reached terminal node value v(z t ) are revealed. To learn NE, the regret minimizer is required to estimate the full-feedback loss gradient t by v(z t ) and minimize the regret. In this paper, we propose a generalized framework for this learning setting. It presents a theoretical framework for the design and the modular analysis of the bandit regret minimization methods. We demonstrate that the most recent bandit regret minimization methods can be analyzed as a particular case of our framework. Following this framework, we describe a novel method SIX-OMD to learn approximate NE. It is model-free and extremely improves the best existing convergence rate from the order of O( XB/T + Y C/T ) 1 to O( M X /T + M Y /T ) 2 . Moreover, SIX-OMD is computationally efficient as it needs to perform the current strategy and average strategy updates only along the sampled trajectory.1 X, Y are the number of information sets and A, B are the number of actions for the two players. 2 MX and MY are defined in Section 5.2, they are extremely less than X and Y .Preprint. Under review.

A Survey of Decision Making in Adversarial Games

“…The authors in [117] recently bridged several standing gaps between NFG and EFG learning by directly transferring desirable properties in NFGs to EFGs, guaranteeing simultaneously last-iterate convergence, lower dependence on the game size, and constant regret in games. Besides, bandit feedback is of practical importance in real-world applications for II-ZSGs [118], [119], where only the interactive trajectory and the payoff of the reached terminal node can be observed without prior knowledge of the game, such as the tree structure, the observation/state space, and transition probabilities (for Markov games) [120]. On the other hand, multi-player II-ZSGs are more challenging and thus have been less researched except for a handful of works, for example, Pluribus [121], the first multi-player poker agent, has defeated top humans in six-player no-limit Texas Hold'em poker (the most prevalent poker in the world) [122], and other endeavors [119], [123]- [126].…”