Online Double Oracle

Dinh, Le Cong; Yang, Yanfei; McAleer, Stephen; Tian, Zheng; Nieves, Nicolas Perez; Oliver, Slumbers,; Mguni, David; Ammar, Haitham Bou; Wang, Jun

doi:10.48550/arxiv.2103.07780

Cited by 6 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These algorithms involving combining singleagent RL algorithm, such as deep Q-network (DQN) [35] or proximal policy optimisation (PPO) [38], with best-response-based Nash equilibrium finding algorithm for normal-form games, including fictitious play [10], double oracle [34], and many others. A few other examples such as neural fictitious self-play (NFSP) [21], policy space response oracles (PSRO) [28], online double oracle [13] and prioritized fictitious self-play [45] also in general fall into this class of algorithms or their variants. These algorithms call single-agent RL algorithm to compute the base response of the current "metastrategy", and then use the best-response-based algorithm for normal-form games to compute a new "meta-strategy".…”

Section: Discussionmentioning

confidence: 99%

A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games

Ding¹,

Su²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper proposes novel, end-to-end deep reinforcement learning algorithms for learning two-player zero-sum Markov games. Our objective is to find the Nash Equilibrium policies, which are free from exploitation by adversarial opponents. Distinct from prior efforts on finding Nash equilibria in extensive-form games such as Poker, which feature tree-structured transition dynamics and discrete state space, this paper focuses on Markov games with general transition dynamics and continuous state space. We propose (1) Nash DQN algorithm, which integrates DQN [35] with a Nash finding subroutine for the joint value functions; and (2) Nash DQN Exploiter algorithm, which additionally adopts an exploiter for guiding agent's exploration. Our algorithms are the practical variants of theoretical algorithms which are guaranteed to converge to Nash equilibria in the basic tabular setting. Experimental evaluation on both tabular examples and two-player Atari games demonstrates the robustness of the proposed algorithms against adversarial opponents, as well as their advantageous performance over existing methods.

show abstract

Section: Discussionmentioning

confidence: 99%

A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games

Ding¹,

Su²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This can be achieved through various means, such as monotonic improvement in exploitability [55], regret bound [64]. Another approach to improving efficiency in PB-DRL is to use distributed computing techniques [43,65]. These techniques can enable faster evaluation of individuals within a population, as well as better parallelization of the learning process.…”

Section: Challengesmentioning

confidence: 99%

A Survey on Population-Based Deep Reinforcement Learning

et al. 2023

View full text Add to dashboard Cite

Many real-world applications can be described as large-scale games of imperfect information, which require extensive prior domain knowledge, especially in competitive or human–AI cooperation settings. Population-based training methods have become a popular solution to learn robust policies without any prior knowledge, which can generalize to policies of other players or humans. In this survey, we shed light on population-based deep reinforcement learning (PB-DRL) algorithms, their applications, and general frameworks. We introduce several independent subject areas, including naive self-play, fictitious self-play, population-play, evolution-based training methods, and the policy-space response oracle family. These methods provide a variety of approaches to solving multi-agent problems and are useful in designing robust multi-agent reinforcement learning algorithms that can handle complex real-life situations. Finally, we discuss challenges and hot topics in PB-DRL algorithms. We hope that this brief survey can provide guidance and insights for researchers interested in PB-DRL algorithms.

show abstract

“…bilinear game is usually denoted by the payoff matrices pair (A, B), which is zero-sum when B = −A, and as an important notion, the rank of a game (A, B) is defined as the rank of matrix A + B. Several interesting games can be viewed as special cases of bilinear games, such as bimatrix games [47]- [49], where…”

Section: A Zero-sum Games (Zsgs)mentioning

confidence: 99%

“…For normal-form games, a large number of algorithms have so far been proposed, e.g., regret matching (RM for short, first proposed by Hart and Mas-Colell in 2000 [220]), RM+ [221], fictitious play [222], [223], double oracle [224], online double oracle [49], and among others. Wherein, the most prevalent algorithms are based on regret learning, usually called noregret (or sublinear) learning algorithms, depending external and internal regrets in general, as defined below.…”

Section: A Zero-sum Normal-and Extensive-form Gamesmentioning

confidence: 99%

A Survey of Decision Making in Adversarial Games

Li¹,

Mao²,

Ye³

et al. 2022

Preprint

View full text Add to dashboard Cite

Game theory has by now found numerous applications in various fields, including economics, industry, jurisprudence, and artificial intelligence, where each player only cares about its own interest in a noncooperative or cooperative manner, but without obvious malice to other players. However, in many practical applications, such as poker, chess, evader pursuing, drug interdiction, coast guard, cyber-security, and national defense, players often have apparently adversarial stances, that is, selfish actions of each player inevitably or intentionally inflict loss or wreak havoc on other players. Along this line, this paper provides a systematic survey on three main game models widely employed in adversarial games, i.e., zero-sum normal-form and extensive-form games, Stackelberg (security) games, zero-sum differential games, from an array of perspectives, including basic knowledge of game models, (approximate) equilibrium concepts, problem classifications, research frontiers, (approximate) optimal strategy seeking techniques, prevailing algorithms, and practical applications. Finally, promising future research directions are also discussed for relevant adversarial games.

show abstract

Online Double Oracle

Cited by 6 publications

References 0 publications

A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games

A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games

A Survey on Population-Based Deep Reinforcement Learning

A Survey of Decision Making in Adversarial Games

Contact Info

Product

Resources

About