Solving Common-Payoff Games with Approximate Policy Iteration

Abstract

For artificially intelligent learning systems to have widespread applicability in real-world settings, it is important that they be able to operate decentrally. Unfortunately, decentralized control is difficult---computing even an epsilon-optimal joint policy is a NEXP complete problem. Nevertheless, a recently rediscovered insight---that a team of agents can coordinate via common knowledge---has given rise to algorithms capable of finding optimal joint policies in small common-payoff games. The Bayesian action decoder (BAD) leverages this insight and deep reinforcement learning to scale to games as large as two-player Hanabi. However, the approximations it uses to do so prevent it from discovering optimal joint policies even in games small enough to brute force optimal solutions. This work proposes CAPI, a novel algorithm which, like BAD, combines common knowledge with deep reinforcement learning. However, unlike BAD, CAPI prioritizes the propensity to discover optimal joint policies over scalability. While this choice precludes CAPI from scaling to games as large as Hanabi, empirical results demonstrate that, on the games to which CAPI does scale, it is capable of discovering optimal joint policies even when other modern multi-agent reinforcement learning algorithms are unable to do so.

Student of Games: A unified learning algorithm for both perfect and imperfect information games

“…Another work used a method inspired by DeepStack applied to The Resistance (13). In the cooperative setting, several works have made use of belief-based learning (and search) using public subgame decomposition (12,61,62), applied to Hanabi (11). Learning and game-theoretic reasoning were also recently combined to produce agents that play well with humans without human data on the collaborative game Overcooked (63).…”

Abstracting Imperfect Information Away from Two-Player Zero-Sum Games

“…As discussed in the background, provided a reduction from solving common-payoff games to solving belief MDPs; independently, Dibangoye et al (2013b) and Oliehoek (2013) discovered similar reductions. These ideas have been leveraged in a large body of work in decentralized control literature (Lessard & Nayyar, 2013;Nayyar et al, 2014;Arabneydi & Mahajan, 2014;Ouyang et al, 2015;Vasconcelos & Martins, 2016;Tavafoghi et al, 2016;Afshari & Mahajan, 2018;Gagrani & Nayyar, 2018;Tavafoghi et al, 2018;Zhang et al, 2019;Gupta, 2021) and machine learning literature (Dibangoye et al, 2013a;MacDermed & Isbell, 2013;Dibangoye et al, 2014a;b;Dibangoye & Buffet, 2018;Foerster et al, 2019;Sokota et al, 2021;Fickinger et al, 2021;Sokota et al, 2022b;Kao et al, 2022). Use cases include game solving (Dibangoye et al, 2013b), expert iteration (Sokota et al, 2021), and decision-time planning Fickinger et al, 2021;Sokota et al, 2022b).…”