Faster First-Order Methods for Extensive-Form Game Solving

Kroer, Christian; Waugh, Kevin; Kılınç-Karzan, Fatma; Sandholm, Tüomas

doi:10.1145/2764468.2764476

Cited by 22 publications

(54 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This allows each term in the expected loss to be weighted only by the sequence ending in the corresponding action. The sequence form has been used to instantiate linear programming (von Stengel, 1996) and firstorder methods (Hoda et al, 2010;Kroer et al, 2015 for computing Nash equilibria of zero-sum EFGs. There is a straightforward mapping between any x ∈ X to its corresponding sequence form: simply assign each sequence the product of probabilities in the sequence.…”

Section: Laminar Regret Decompositionmentioning

confidence: 99%

“…Quantal response equilibrium (QRE) Ling, Fang, and Kolter (2018) show that a reduced-normal-form QRE can be expressed as the convex-concave saddle-point problem (6) where d 1 and d 2 are the (convex) dilated entropy functions usually used in first-order methods (FOMs) for solving EFGs (Hoda et al, 2010;Kroer et al, 2015. This saddle-point problem can be solved using FOMs, which would lead to fast convergence rate due to the strongly convex nature of the dilated entropy distance .…”

Section: Extensive-form Games With Convex-concave Saddle-point Structurementioning

confidence: 99%

“…If we apply a convex regularization function at each simplex, we can apply our framework to solve the resulting game. For the negative entropy regularizer this is equivalent to the dilated entropy distance function used for solving EFGs with firstorder methods (Hoda et al, 2010;Kroer et al, 2015. Ling, Fang, and Kolter (2018) show that dilated-entropyregularized EFGs are equivalent to quantal response equilibria (QRE) in the corresponding reduced normal-form game.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games

Farina

Kroer

Sandholm

2019

AAAI

Self Cite

View full text Add to dashboard Cite

Regret minimization is a powerful tool for solving large-scale extensive-form games. State-of-the-art methods rely on minimizing regret locally at each decision point. In this work we derive a new framework for regret minimization on sequential decision problems and extensive-form games with general compact convex sets at each decision point and general convex losses, as opposed to prior work which has been for simplex decision points and linear losses. We call our framework laminar regret decomposition. It generalizes the CFR algorithm to this more general setting. Furthermore, our framework enables a new proof of CFR even in the known setting, which is derived from a perspective of decomposing polytope regret, thereby leading to an arguably simpler interpretation of the algorithm. Our generalization to convex compact sets and convex losses allows us to develop new algorithms for several problems: regularized sequential decision making, regularized Nash equilibria in extensive-form games, and computing approximate extensive-form perfect equilibria. Our generalization also leads to the first regret-minimization algorithm for computing reduced-normal-form quantal response equilibria based on minimizing local regrets. Experiments show that our framework leads to algorithms that scale at a rate comparable to the fastest variants of counterfactual regret minimization for computing Nash equilibrium, and therefore our approach leads to the first algorithm for computing quantal response equilibria in extremely large games. Finally we show that our framework enables a new kind of scalable opponent exploitation approach.

show abstract

Section: Laminar Regret Decompositionmentioning

confidence: 99%

Section: Extensive-form Games With Convex-concave Saddle-point Structurementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games

Farina

Kroer

Sandholm

2019

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hoda et al [2010] initially proposed DGFs for EFGs leading to O( 1 ) convergence rate when used with EGT. Kroer et al [2015] improved these result for the dilated entropy function. Gilpin et al [2012] give an algorithm with convergence rate O(ln( 1 )).…”

Section: Related Workmentioning

confidence: 79%

“…In this paper we focus on the second approach. Iterative game solvers mainly fall in two categories: (i) counterfactual-regret-based methods [Zinkevich et al, 2007, Lanctot et al, 2009] achieving a convergence rate on the order of O( 1 2 ), and (ii) first-order methods (FOMs) [Hoda et al, 2010, Kroer et al, 2015 achieving a convergence rate of O( 1 ). The better convergence rate of FOMs makes them more attractive from a theoretical viewpoint.…”

Section: Introductionmentioning

confidence: 99%

Theoretical and Practical Advances on Smoothing for Extensive-Form Games

Kroer

Waugh

Kılınç-Karzan

et al. 2017

Proceedings of the 2017 ACM Conference on Economics and Computation

Self Cite

View full text Add to dashboard Cite

Sparse iterative methods, in particular first-order methods, are known to be among the most effective in solving large-scale two-player zero-sum extensive-form games. The convergence rates of these methods depend heavily on the properties of the distance-generating function that they are based on. We investigate the acceleration of first-order methods for solving extensive-form games through better design of the dilated entropy function-a class of distance-generating functions related to the domains associated with the extensive-form games. By introducing a new weighting scheme for the dilated entropy function, we develop the first distance-generating function for the strategy spaces of sequential games that only a logarithmic dependence on the branching factor of the player. This result improves the convergence rate of several first-order methods by a factor of Ω(b d d), where b is the branching factor of the player, and d is the depth of the game tree.Thus far, counterfactual regret minimization methods have been faster in practice, and more popular, than first-order methods despite their theoretically inferior convergence rates. Using our new weighting scheme and practical tuning we show that, for the first time, the excessive gap technique can be made faster than the fastest counterfactual regret minimization algorithm, CFR+, in practice.

show abstract