Polygames: Improved zero learning

Abstract

Since DeepMind’s AlphaZero, Zero learning quickly became the state-of-the-art method for many board games. It can be improved using a fully convolutional structure (no fully connected layer). Using such an architecture plus global pooling, we can create bots independent of the board size. The training can be made more robust by keeping track of the best checkpoints during the training and by training against them. Using these features, we release Polygames, our framework for Zero learning, with its library of games and its checkpoints. We won against strong humans at the game of Hex in 19 × 19, including the human player with the best ELO rank on LittleGolem; we incidentally also won against another Zero implementation, which was weaker than humans: in a discussion on LittleGolem, Hex19 was said to be intractable for zero learning. We also won in Havannah with size 8: win against the strongest player, namely Eobllor, with excellent opening moves. We also won several first places at the TAAI 2019 competitions and had positive results against strong bots in various games.

Biasing MCTS with Features for General Games

“…In recent years, this is often based on combinations of Monte-Carlo Tree Search (MCTS) (Kocsis & Szepesvári, 2006;Coulom, 2007b;Browne et al, 2012) and deep neural networks (DNNs) (LeCun, Bengio, & Hinton, 2015). In principle this combination of techniques can be successfully applied (Silver et al, 2016;Anthony et al, 2017;Silver et al, 2017;Lorentz & Zosa IV, 2017;Silver et al, 2018;Tian et al, 2019;Morandin et al, 2019;Wu, 2019;Cazenave et al, 2020;Cazenave, 2020) to a wide variety of games. However, in practice the high computational requirements make it infeasible to scale this up to large-scale studies that involve training agents for hundreds or thousands of distinct games (Stephenson, Crist, & Browne, 2020), in addition to possibly many more variants of games generated automatically as possible reconstructions of games with incomplete rules (Browne, 2018;Browne et al, 2019b).…”

Deep learning for general game playing with Ludii and Polygames

“…Self-play training approaches such as those popularised by AlphaGo Zero (Silver et al, 2017) and AlphaZero (Silver et al, 2018), based on combinations of Monte-Carlo tree search (MCTS) (Kocsis and Szepesvári, 2006;Coulom, 2007;Browne et al, 2012) and Deep Learning (LeCun et al, 2015), have been demonstrated to be fairly generally applicable, and achieved state-of-the-art results in a variety of board games such as Go (Silver et al, 2017), Chess, Shogi (Silver et al, 2018), Hex, and Havannah (Cazenave et al, 2020). These approaches require relatively little domain knowledge, but still require some in the form of:…”

“…In this paper, we describe how we combine the GGP system Ludii and the PyTorch-based (Paszke et al, 2019) state-of-the-art training algorithms in Polygames (Cazenave et al, 2020), with the goal of mitigating all three of the requirements for domain knowledge listed above. Section 2 provides some background information on these training techniques.…”

“…In addition to such typical architectures, Polygames (Cazenave et al, 2020) includes various different structures, such as:…”

Descent wins five gold medals at the Computer Olympiad

“…• Breakthrough: DaSoJai (author: Wei-Lin Wu and Shun-Shii Lin) and Polygames (Facebook NDHU). Polygames (Cazenave et al, 2020) is a reimplementation of AlphaZero. • Amazons: 8QP (Johan de Koning) and SherlockGo (Liang Shuang, Liang Tailin, Wang Jilong, Li Xiaorui, and Zhou Ke).…”