Rethinking Formal Models of Partially Observable Multiagent Decision Making

Kovařík, Vojtěch; Schmid, Martin; Burch, Neil; Bowling, Michael; Lisý, Viliam

doi:10.48550/arxiv.1906.11110

Cited by 7 publications

(9 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Common-Payoff FOSGs To describe GME clearly, we require a notational distinction between information that is known to all players and information that is not. Toward that end, we adapt our notation from the factored observation stochastic game (FOSG) formalism [18]. FOSGs closely resemble partially observable stochastic games (POSGs) [14] but take a more explicit approach to public knowledge.…”

Section: Notationmentioning

confidence: 99%

Communicating via Markov Decision Processes

Sokota¹,

Witt²,

Igl³

et al. 2021

Preprint

View full text Add to dashboard Cite

In many common-payoff games, achieving good performance requires players to develop protocols for communicating their private information implicitly-i.e., using actions that have non-communicative effects on the environment. Multi-agent reinforcement learning practitioners typically approach this problem using independent learning methods in the hope that agents will learn implicit communication as a byproduct of expected return maximization. Unfortunately, independent learning methods are incapable of doing this in many settings. In this work, we isolate the implicit communication problem by identifying a class of partially observable common-payoff games, which we call implicit referential games, whose difficulty can be attributed to implicit communication. Next, we introduce a principled method based on minimum entropy coupling that leverages the structure of implicit referential games, yielding a new perspective on implicit communication. Lastly, we show that this method can discover performant implicit communication protocols in settings with very large spaces of messages. * Equal contribution (order determined by coin flip).Preprint. Under review.

show abstract

Section: Notationmentioning

confidence: 99%

Communicating via Markov Decision Processes

Sokota¹,

Witt²,

Igl³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Our notation is based on that of factored observation games [36] which is a modification of partially observable stochastic games [27] that distinguishes between private and public observations. We consider a game with N = {1, 2, ..., N } agents.…”

Section: Notation and Backgroundmentioning

confidence: 99%

“…As explained in[36], it may be possible for agents to infer common knowledge beyond just public observations. However, doing this additional reasoning is inefficient both theoretically and practically.…”

mentioning

confidence: 99%

Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

Brown,

Bakhtin,

Lerer

et al. 2020

Preprint

View full text Add to dashboard Cite

The combination of deep reinforcement learning and search at both training and test time is a powerful paradigm that has led to a number of a successes in singleagent settings and perfect-information games, best exemplified by the success of AlphaZero. However, algorithms of this form have been unable to cope with imperfect-information games. This paper presents ReBeL, a general framework for self-play reinforcement learning and search for imperfect-information games. In the simpler setting of perfect-information games, ReBeL reduces to an algorithm similar to AlphaZero. Results show ReBeL leads to low exploitability in benchmark imperfect-information games and achieves superhuman performance in heads-up no-limit Texas hold'em poker, while using far less domain knowledge than any prior poker AI. We also prove that ReBeL converges to a Nash equilibrium in two-player zero-sum games in tabular settings. * Equal contribution Preprint. Under review.

show abstract

“…3 EFGs are a standard model of sequential decision-making, described, for example, in (Osborne and Rubinstein, 1994). Partiallyobservable stochastic games (Hansen, Bernstein, and Zilberstein, 2004) constitute an equally valid (Kovařík et al, 2019) 7. Debate error is the resulting deviation τ (q, w, o(w, t))…”

Section: The Debate Environmentmentioning

confidence: 99%

(When) Is Truth-telling Favored in AI Debate?

Kovařík,

Carey

2019

Preprint

Self Cite

View full text Add to dashboard Cite

For some problems, it is difficult for humans to judge the goodness of AI-proposed solutions. Irving, Christiano, and Amodei (2018) propose that in such cases, we may use a debate between two AI systems to assist the human judge to select a good answer. We introduce a mathematical framework for modelling this type of debate and propose that the quality of debate designs may be measured by the accuracy of the most persuasive answer. We describe a simple instance of the debate framework called feature debate and analyze the degree to which such debates track the truth. We argue that despite being very simple, feature debates capture many aspects of practical debates such as the incentives to confuse the judge or stall to prevent losing. We analyze two special types of debates, those where arguments constitute independent evidence about the topic, and those where the information bandwidth of the judge is limited. * We are grateful to Chris van Merwijk, Lukas Finnveden, Michael Cohen, and Michael Dennis for feedback and discussions related to this text. The first author was partially supported by MEYS funded project CZ.02.1.01/0.0/0.0/16 019/0000765 "Research Center for Informatics".

show abstract

Rethinking Formal Models of Partially Observable Multiagent Decision Making

Cited by 7 publications

References 18 publications

Communicating via Markov Decision Processes

Communicating via Markov Decision Processes

Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

(When) Is Truth-telling Favored in AI Debate?

Contact Info

Product

Resources

About