Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

Silver, David; Hubert, Thomas; Schrittwieser, Julian; Antonoglou, Ioannis; Lai, Matthew; Guez, Arthur; Lanctot, Marc; Sifre, Laurent; Kumaran, Dharshan; Graepel, Thore; Lillicrap, Timothy; Simonyan, Karen; Hassabis, Demis

doi:10.48550/arxiv.1712.01815

Cited by 318 publications

(427 citation statements)

References 7 publications

Supporting

Mentioning

424

Contrasting

Unclassified

Order By: Relevance

“…Deep Reinforcement Learning (RL) has gained many successes against humans in competitive games, such as Go [41], Dota [31], and StarCraft [48]. However, it still remains a challenge to build AI agents that can coordinate and collaborate with humans that the agents have not seen during training [20,24,5,40,18,21].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Zhao¹,

Song²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

An AI agent should be able to coordinate with humans to solve tasks. We consider the problem of training a Reinforcement Learning (RL) agent without using any human data, i.e., in a zero-shot setting, to make it capable of collaborating with humans. Standard RL agents learn through self-play. Unfortunately, these agents only know how to collaborate with themselves and normally do not perform well with unseen partners, such as humans. The methodology of how to train a robust agent in a zero-shot fashion is still subject to research. Motivated from the maximum entropy RL, we derive a centralized population entropy objective to facilitate learning of a diverse population of agents, which is later used to train a robust agent to collaborate with unseen partners. The proposed method shows its effectiveness compared to baseline methods, including self-play PPO, the standard Population-Based Training (PBT), and trajectory diversity-based PBT, in the popular Overcooked game environment. We also conduct online experiments with real humans and further demonstrate the efficacy of the method in the real world. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.

show abstract

Section: Introductionmentioning

confidence: 99%

“…The mainstream method for building state-of-the-art AI agents is through self-play RL [44,41]. Self-play-trained agents are very specialized, and therefore suffered significantly from distributional shift when paired with humans.…”

Section: Introductionmentioning

confidence: 99%

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Zhao¹,

Song²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Over the last few years, RL, grounded on combining classical theoretical results with deep learning and the functional approximation paradigm, has proved to be a fruitful approach to many artificial intelligence tasks from diverse domains. Breakthrough achievements include reaching human-level performance in such complex games as Go [184], and multi-player StarCraft II [206]. The generality of the reinforcement learning framework allows its application in both discrete and continuous spaces to solve tasks in both real and simulated environments [138].…”

Section: The Basics Of Reinforcement Learningmentioning

confidence: 99%

Recent Advances in Reinforcement Learning in Finance

Hambly,

Xu,

Yang

2021

Preprint

View full text Add to dashboard Cite

The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision-making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value and policy based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. Our survey concludes by discussing the application of these RL algorithms in a variety of decision-making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo-advising.

show abstract

“…The rule scheduling problem could be formulated as a reinforcement learning problem where the rule scheduler (the agent) in a given e-graph and its history (the environment) has to decide at a given iteration if a rule should be applied or not (the possible actions). Game mastering reinforcement learning techniques such as AlphaZero [49] or MuZero [47] could be adopted by viewing the rule scheduling problem as a game that the agent can win, especially in an automated theorem proving context. Viewing e-graph rule scheduling as a reinforcement learning task is thus a very interesting open research problem that, together with a proof production algorithm (subsection 4.1.2), may yield groundbreaking results in many applications such as automated theorem proving (subsection 4.3.1), compiler optimizations and symbolic mathematics whereas equality saturation with classical heuristics may take a very long time to optimize and prove equalities in the presence of very large input expressions and systems of equational rewrite rules.…”

Section: Smart Rule Scheduling Heuristicsmentioning

confidence: 99%