Scalability and Parallelization of Monte-Carlo Tree Search

Bourki, Amine; Chaslot, Guillaume; Coulm, Matthieu; Danjean, Vincent; Doghmen, Hassen; Hoock, Jean‐Baptiste; Hérault, Thomas; Rimmel, Arpad; Teytaud, Fabien; Teytaud, Olivier; Vayssière, Paul; Yu, Zhiyi

doi:10.1007/978-3-642-17928-0_5

Cited by 32 publications

(27 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We started with the UCB algorithm and this lead to the so-called UCT (Upper Confidence Bounds applied to Trees) algorithm, which was independently developed and analyzed by Csaba Szepesvári and Levente Kocsis [78]. Several major improvements (such as the use of features in the random playouts, the Rapid Action Value Estimation (RAVE), the parallelization of the algorithm, and the introduction of opening books) [56,91,21,97,43,57] …”

Section: Historical Motivation In Computer Gomentioning

confidence: 99%

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Munos

2014

FNT in Machine Learning

197

181

View full text Add to dashboard Cite

This work covers several aspects of the optimism in the face of uncertainty principle applied to large scale optimization problems under finite numerical budget. The initial motivation for the research reported here originated from the empirical success of the so-called Monte-Carlo Tree Search method popularized in Computer Go and further extended to many other games as well as optimization and planning problems. Our objective is to contribute to the development of theoretical foundations of the field by characterizing the complexity of the underlying optimization problems and designing efficient algorithms with performance guarantees. The main idea presented here is that it is possible to decompose a complex decision making problem (such as an optimization problem in a large search space) into a sequence of elementary decisions, where each decision of the sequence is solved using a (stochastic) multi-armed bandit (simple mathematical model for decision making in stochastic environments). This so-called hierarchical bandit approach (where the reward observed by a bandit in the hierarchy is itself the return of another bandit at a deeper level) possesses the nice feature of starting the exploration by a quasi-uniform sampling of the space and then focusing progressively on the most promising area, at different scales, according to the evaluations observed so far, and eventually performing a local search around the global optima of the function. The performance of the method is assessed in terms of the optimality of the returned solution as a function of the number of function evaluations. Our main contribution to the field of function optimization is a class of hierarchical optimistic algorithms designed for general search spaces (such as metric spaces, trees, graphs, Euclidean spaces, ...) with different algorithmic instantiations depending on whether the evaluations are noisy or noiseless and whether some measure of the "smoothness" of the function is known or unknown. The performance of the algorithms depends on the local behavior of the function around its global optima expressed in terms of the quantity of near-optimal states measured with some metric. If this local smoothness of the function is known then one can design very efficient optimization algorithms (with convergence rate independent of the space dimension), and when it is not known, we can build adaptive techniques that can, in some cases, perform almost as well as when it is known. In order to be self-contained, we start with a brief introduction to the stochastic multi-armed bandit problem in Chapter 1 and describe the UCB (Upper Confidence Bound) strategy and several extensions. In Chapter 2 we present the Monte-Carlo Tree Search method applied to Computer Go and show the limitations of previous algorithms such as UCT (UCB applied to Trees). This provides motivation for designing theoretically well-founded optimistic optimization algorithms. The main contributions on hierarchical optimistic optimization are described in Chapters 3 and 4...

show abstract

Section: Historical Motivation In Computer Gomentioning

confidence: 99%

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Munos

2014

FNT in Machine Learning

197

181

View full text Add to dashboard Cite

show abstract

“…When information is shared, only a portion of the tree is shared in order to minimise the communication overhead of sharing. A possible sharing strategy is to share the nodes in the top 3 levels that have at least 5% of the total playouts through them, at a frequency of 3 Hz [21]. In this method, each of the processing nodes performs all four steps of the MCTS iterations on its tree.…”

Section: Root Parallelisationmentioning

confidence: 99%

“…If root parallelisation could update at infinite frequency, share the whole tree, and update in zero time, then it would be equivalent to tree parallelisation. Bourki et al have shown root parallelisation to scale to 40 nodes [21]. [23].…”

Section: Root Parallelisationmentioning

confidence: 99%

“…Leaf and root parallelisation are therefore the only realistic candidates for cluster parallelisation. Root parallelisation, as described in Section 2.5.4, was chosen due to it scaling better than leaf parallelisation in previous work [21,25].…”

Section: Cluster Parallelisationmentioning

confidence: 99%

“…The portion of the tree shared is based on the top few tree levels and how many playouts have passed through the node; this approach was chosen based on other cluster root parallelisation implementations [21]. The exact number of levels and playouts that signify the shared portion are adjustable parameters.…”

Section: Cluster Parallelisationmentioning

confidence: 99%

See 2 more Smart Citations

Monte-Carlo tree search parallelisation for computer go

Niekerk

Rooyen

Kroon

et al. 2012

Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference

View full text Add to dashboard Cite

Parallelisation of computationally expensive algorithms, such as Monte-Carlo Tree Search (MCTS), has become increasingly important in order to increase algorithm performance by making use of commonplace parallel hardware.Oakfoam, an MCTS-based Computer Go player, was extended to support parallel processing on multi-core and cluster systems. This was done using tree parallelisation for multi-core systems and root parallelisation for cluster systems.Multi-core parallelisation scaled linearly on the tested hardware on 9x9 and 19x19 boards when using the virtual loss modification. Cluster parallelisation showed poor results on 9x9 boards, but scaled well on 19x19 boards, where it achieved a four-node ideal strength increase on eight nodes.Due to this work, Oakfoam is currently one of only two open-source MCTS-based Computer Go players with cluster parallelisation, and the only one using the Message Passing Interface (MPI) standard.

show abstract