The Google DeepMind challenge match in March 2016 was a historic achievement for computer Go development. This article discusses the development of computational intelligence (CI) and its relative strength in comparison with human intelligence for the game of Go. We first summarize the milestones achieved for computer Go from 1998 to 2016. Then, the computer Go programs that have participated in previous IEEE CIS competitions as well as methods and techniques used in AlphaGo are briefly introduced. Commentaries from three high-level professional Go players on the five AlphaGo versus Lee Sedol games are also included. We conclude that AlphaGo beating Lee Sedol is a huge achievement in artificial intelligence (AI) based largely on CI methods. In the future, powerful computer Go programs such as AlphaGo are expected to be instrumental in promoting Go education and AI real-world applications.

This paper proposes a new approach to a novel value network architecture for the game Go, called a multi-labelled (ML) value network. In the ML value network, different values (win rates) are trained simultaneously for different settings of komi, a compensation given to balance the initiative of playing first. The ML value network has three advantages, (a) it outputs values for different komi, (b) it supports dynamic komi, and (c) it lowers the mean squared error (MSE). This paper also proposes a new dynamic komi method to improve game-playing strength.This paper also performs experiments to demonstrate the merits of the architecture. First, the MSE of the ML value network is generally lower than the value network alone. Second, the program based on the ML value network wins by a rate of 67.6% against the program based on the value network alone. Third, the program with the proposed dynamic komi method significantly improves the playing strength over the baseline that does not use dynamic komi, especially for handicap games. To our knowledge, up to date, no handicap games have been played openly by programs using value networks. This paper provides these programs with a useful approach to playing handicap games.Although the rules of Go are simple, its game tree complexity is extremely high, estimated to be 10 360 in [1] [40]. It is common for players with different strengths to play ℎ-stone handicap games, where the weaker player, usually designated to play as black, is allowed to place ℎ stones 2 first with a komi of 0.5 before white makes the first move. If the strength difference (rank difference) between both players is large, more handicap stones are usually given to the weaker player.In the past, computer Go was listed as one of the AI grand challenges [16][28]. By 2006, the strengths of computer Go programs were generally below 6 kyu [5][8][14], far away from amateur dan players. In 2006, Monte Carlo tree search (MCTS) [6][11][15][23][37] was invented and computer Go programs started making significant progress [4][10][13], roughly up to 6 dan in 2015. In 2016, this grand challenge was achieved by the program AlphaGo [34] when it defeated (4:1) Lee Sedol, a 9 dan grandmaster who had won the most world Go champion titles in the past decade. Many thought at the time there would be a decade or more away from surpassing this milestone. Up to date, DeepMind, the team behind AlphaGo, had published the techniques and methods of AlphaGo in Nature [34]. AlphaGo was able to surpass experts' expectations by proposing a new method that uses three deep convolutional neural networks (DCNNs) [24][25]: a supervised learning (SL) policy network [7][9][18][26][38] learning to predict experts' moves from human expert game records, a reinforcement learning (RL) policy network [27] improving the SL policy network via self-play, and a value network that performs state evaluation based on self-play game simulations. AlphaGo then combined the DCNNs with MCTS for move generation during game play. In MCTS, a fast rollout policy was...

No abstract

Many of the strongest game playing programs use a combination of Monte Carlo tree search (MCTS) and deep neural networks (DNN), where the DNNs are used as policy or value evaluators. Given a limited budget, such as online playing or during the self-play phase of AlphaZero (AZ) training, a balance needs to be reached between accurate state estimation and more MCTS simulations, both of which are critical for a strong game playing agent. Typically, larger DNNs are better at generalization and accurate evaluation, while smaller DNNs are less costly, and therefore can lead to more MCTS simulations and bigger search trees with the same budget. This paper introduces a new method called the multiple policy value MCTS (MPV-MCTS), which combines multiple policy value neural networks (PV-NNs) of various sizes to retain advantages of each network, where two PV-NNs f S and f L are used in this paper. We show through experiments on the game NoGo that a combined f S and f L MPV-MCTS outperforms single PV-NN with policy value MCTS, called PV-MCTS. Additionally, MPV-MCTS also outperforms PV-MCTS for AZ training.

AlphaZero has been very successful in many games. Unfortunately, it still consumes a huge amount of computing resources, the majority of which is spent in self-play. Hyperparameter tuning exacerbates the training cost since each hyperparameter configuration requires its own time to train one run, during which it will generate its own self-play records. As a result, multiple runs are usually needed for different hyperparameter configurations. This paper proposes using population based training (PBT) to help tune hyperparameters dynamically and improve strength during training time. Another significant advantage is that this method requires a single run only, while incurring a small additional time cost, since the time for generating self-play records remains unchanged though the time for optimization is increased following the AlphaZero training algorithm. In our experiments for 9x9 Go, the PBT method is able to achieve a higher win rate for 9x9 Go than the baselines, each with its own hyperparameter configuration and trained individually. For 19x19 Go, with PBT, we are able to obtain improvements in playing strength. Specifically, the PBT agent can obtain up to 74% win rate against ELF OpenGo, an open-source state-of-the-art AlphaZero program using a neural network of a comparable capacity. This is compared to a saturated non-PBT agent, which achieves a win rate of 47% against ELF OpenGo under the same circumstances.