Non-Asymptotic Analysis of Monte Carlo Tree Search

Abstract

In this work, we consider the popular tree-based search strategy within the framework of reinforcement learning, the Monte Carlo Tree Search (MCTS), in the context of infinite-horizon discounted cost Markov Decision Process (MDP) with deterministic transitions. While MCTS is believed to provide an approximate value function for a given state with enough simulations, cf. [5,6], the claimed proof of this property is incomplete. This is due to the fact that the variant of MCTS, the Upper Confidence Bound for Trees (UCT), analyzed in prior works utilizes "logarithmic" bonus term for balancing exploration and exploitation within the tree-based search, following the insights from stochastic multi-arm bandit (MAB) literature, cf. [1,3]. In effect, such an approach assumes that the regret of the underlying recursively dependent non-stationary MABs concentrates around their mean exponentially in the number of steps, which is unlikely to hold as pointed out in [2], even for stationary MABs.As the key contribution of this work, we establish polynomial concentration property of regret for a class of non-stationary multiarm bandits. This in turn establishes that the MCTS with appropriate polynomial rather than logarithmic bonus term in UCB has the claimed property of [5,6]. Interestingly enough, empirically successful approaches (cf. [10]) utilize a similar polynomial form of MCTS as suggested by our result. Using this as a building block, we argue that MCTS, combined with nearest neighbor supervised learning, acts as a "policy improvement" operator, i.e., it iteratively improves value function approximation for all states, due to combining with supervised learning, despite evaluating at only finitely many states. In effect, we establish that to learn an ε-approximation of the value function for deterministic MDPs with respect to ℓ ∞ norm, MCTS combined with nearest neighbor requires a sample size scaling as O ε −(d+4) , where d is the dimension of the state space. This is nearly optimal due to a minimax lower bound of Ω ε −(d+2) [8] suggesting the strength of the variant of MCTS we propose here and our resulting analysis. 1

Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning

“…When the base algorithm has convergence guarantees, such as UCRL, we can additionally provide guarantees on the rate of convergence. We provide these rates and a discussion of the UCRL case in the appendix: our analysis drawns upon the analysis of convergence rates for Monte Carlo Tree Search from Shah et al (Shah, Xie, and Xu 2020).…”

“…In the case where the learning algorithms under constraints have regret guarantees, such as UCRL, we can closely follow the techniques of Shah et al (Shah, Xie, and Xu 2020) to provide a concentration property. In this subsection we will first show this generally and then provide discussion for UCRL.…”

Neural Tree Expansion for Multi-Robot Planning in Non-Cooperative Environments

“…|I| k ] that maps joint observation, z, to joint action, a. Adapting the proof concept in [6] to our setting, the policy improvement can be shown by validating the following two properties and then iterating: (11) and, after generating an appropriate dataset, (ii) learning…”

“…MCTS frequently uses the Upper Confidence Bound for Trees algorithm [4] that uses a discrete-action, multi-armed bandit solution [5] to balance exploration and exploitation in node selection. Recent work uses a non-stationary bandit analysis to propose a polynomial, rather than logarithmic, exploration term [6].…”

“…By iterating over these two steps, the model improves over time. The first theoretical analysis of this powerful method is recently shown for single-agent discrete action space problems [6]. In comparison, our method is applied to a continuous state-action, multi-agent setting.…”