Problems with the EFG formalism: a solution attempt using observations

Kovařík, Vojtěch; Lisý, Viliam

doi:10.48550/arxiv.1906.06291

Cited by 2 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 4: A game where (i) and (ii) from Proposition D.1 are not equivalent (for h 0 := the green player's node). (Kovarik and Lisy, 2019) The corollary of Proposition D.1 is that we can safely view (and search trough) subgames on the level of histories, information states, or public states, and all these views are equivalent. This result doesn't hold in EFGs since some of them might fail the implication (ii) =⇒ (i) (consider the EFG from Figure 4 with "public states = histories at the same level").…”

Section: The Identity Mappings H ∈ Hmentioning

confidence: 99%

“…For an example and further details which prove this claim, as well as the proofs of the remaining results, see Appendix E or (Kovarik and Lisy, 2019). In light of this negative result, it is natural to try and come up with a definition which would avoid these shortcomings.…”

Section: Sub-game Decompositionmentioning

confidence: 99%

“…We claim that instead of starting with information sets, it is more intuitive (and technically convenient) to derive information sets from observations. For further details regarding observations in EFGs and related counterexamples, see (Kovarik and Lisy, 2019).…”

Section: E Problems With Decomposition In Efgs: Details and Proofsmentioning

confidence: 99%

See 2 more Smart Citations

Rethinking Formal Models of Partially Observable Multiagent Decision Making

Kovařík¹,

Schmid²,

Burch³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Multiagent decision-making problems in partially observable environments are usually modeled as either extensive-form games (EFGs) within the game theory community or partially observable stochastic games (POSGs) within the reinforcement learning community. While most practical problems can be modeled in both formalisms, the communities using these models are mostly distinct with little sharing of ideas or advances. The last decade has seen dramatic progress in algorithms for EFGs, mainly driven by the challenge problem of poker. We have seen computational techniques achieving super-human performance, some variants of poker are essentially solved, and there are now sound local search algorithms which were previously thought impossible. While the advances have garnered attention, the fundamental advances are not yet understood outside the EFG community. This can be largely explained by the starkly different formalisms between the game theory and reinforcement learning communities and, further, by the unsuitability of the original EFG formalism to make the ideas simple and clear. This paper aims to address these hindrances, by advocating a new unifying formalism, a variant of POSGs, which we call Factored-Observation Games (FOGs). We prove that any timeable perfect-recall EFG can be efficiently modeled as a FOG as well as relating FOGs to other existing formalisms. Additionally, a FOG explicitly identifies the public and private components of observations, which is fundamental to the recent EFG breakthroughs. We conclude by presenting the two building-blocks of these breakthroughs -counterfactual regret minimization and public state decomposition -in the new formalism, illustrating our goal of a simpler path for sharing recent advances between game theory and reinforcement learning community.

show abstract

Section: The Identity Mappings H ∈ Hmentioning

confidence: 99%

Section: Sub-game Decompositionmentioning

confidence: 99%

See 1 more Smart Citation

Rethinking Formal Models of Partially Observable Multiagent Decision Making

Kovařík¹,

Schmid²,

Burch³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We just need to track the amount of money each player has in the pot, the public board cards, and whether there were any bets in the current round 4. Past work defines a subgame to be rooted at a public state[17,9,43,42,12,35,36,57,52]. We break with this definition because imperfect-information subgames rooted at a public state do not have well-defined values.…”

mentioning

confidence: 99%

Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

Brown,

Bakhtin,

Lerer

et al. 2020

Preprint

View full text Add to dashboard Cite

The combination of deep reinforcement learning and search at both training and test time is a powerful paradigm that has led to a number of a successes in singleagent settings and perfect-information games, best exemplified by the success of AlphaZero. However, algorithms of this form have been unable to cope with imperfect-information games. This paper presents ReBeL, a general framework for self-play reinforcement learning and search for imperfect-information games. In the simpler setting of perfect-information games, ReBeL reduces to an algorithm similar to AlphaZero. Results show ReBeL leads to low exploitability in benchmark imperfect-information games and achieves superhuman performance in heads-up no-limit Texas hold'em poker, while using far less domain knowledge than any prior poker AI. We also prove that ReBeL converges to a Nash equilibrium in two-player zero-sum games in tabular settings. * Equal contribution Preprint. Under review.

show abstract

Problems with the EFG formalism: a solution attempt using observations

Cited by 2 publications

References 0 publications

Rethinking Formal Models of Partially Observable Multiagent Decision Making

Rethinking Formal Models of Partially Observable Multiagent Decision Making

Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

Contact Info

Product

Resources

About