Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates

Abstract

In recent years, state-of-the-art game-playing agents often involve policies that are trained in self-playing processes where Monte Carlo tree search (MCTS) algorithms and trained policies iteratively improve each other. The strongest results have been obtained when policies are trained to mimic the search behaviour of MCTS by minimising a cross-entropy loss. Because MCTS, by design, includes an element of exploration, policies trained in this manner are also likely to exhibit a similar extent of exploration. In this paper, we are interested in learning policies for a project with future goals including the extraction of interpretable strategies, rather than state-of-the-art game-playing performance. For these goals, we argue that such an extent of exploration is undesirable, and we propose a novel objective function for training policies that are not exploratory. We derive a policy gradient expression for maximising this objective function, which can be estimated using MCTS value estimates, rather than MCTS visit counts. We empirically evaluate various properties of resulting policies, in a variety of board games.

Monte Carlo Tree Search: A Review of Recent Modifications and Applications

“…Consequently, a computationally heavy process is run just once (offline) and then this time-efficient problem representation can be used in subsequent online applications. The approach of combining an MCTS trainer with a fast learning-based representation can be hybridised in various ways, specific to particular problem / domain of interest (Guo et al, 2014;Kartal et al, 2019a;Soemers et al, 2019).…”

“…-where P (m i ) is the output from the neural network trained on human data; C BT is a weight of how the bias blends with the UCT score and K is a parameter controlling the rate at which the bias decreases. Soemers et al (2019) show that is possible to learn a policy in a MPD using the policy gradient method and value estimates directly from the MCTS algorithm. Kartal et al (2019a) propose a method to combine deep reinforcement learning and MCTS, where the latter acts as a demonstrator for the RL component.…”

Monte Carlo Tree Search: a review of recent modifications and applications

“…It is based on MCTS in which the simulation phase is replaced by a deep RL model that acts in the environment and chooses actions according to its policy rather than randomly. Soemers et al (2019) show that is possible to learn a policy in a MPD using the policy gradient method and value estimates directly from the MCTS algorithm. propose a method to combine deep RL and MCTS, where the latter acts as a demonstrator for the RL component.…”

BRExIt: On Opponent Modelling in Expert Iteration

“…Therefore, the first 10 moves will contain MCTS's exploration, and the rest will feature only the most-visited action. We note that there exists research on this area, with a focus on removing exploration elements from MCTS policy targets with the hope of aiding interpretability [32].…”