Continuous Upper Confidence Trees

Abstract

Abstract. Upper Confidence Trees are a very efficient tool for solving Markov Decision Processes; originating in difficult games like the game of Go, it is in particular surprisingly efficient in high dimensional problems. It is known that it can be adapted to continuous domains in some cases (in particular continuous action spaces). We here present an extension of Upper Confidence Trees to continuous stochastic problems. We (i) show a deceptive problem on which the classical Upper Confidence Tree approach does not work, even with arbitrarily large computational power and with progressive widening (ii) propose an improvement, termed double-progressive widening, which takes care of the compromise between variance (we want infinitely many simulations for each action/state) and bias (we want sufficiently many nodes to avoid a bias by the first nodes) and which extends the classical progressive widening (iii) discuss its consistency and show experimentally that it performs well on the deceptive problem and on experimental benchmarks. We guess that the double-progressive widening trick can be used for other algorithms as well, as a general tool for ensuring a good bias/variance compromise in search algorithms.

Continuous Upper Confidence Trees with Polynomial Exploration – Consistency

“…We have shown the consistency of our modified version, with polynomial exploration and double progressive widening, for a more general case. [6] have shown that the classical UCT is not consistent in this case and already proposed double progressive widening; we here give a proof of the consistency of this approach, when we use polynomial exploration; [6] was using logarithmic exploration.…”

“…For a random node z, we actually have the same property, depending on the double progressive widening constant α R d : this is the so-called double progressive widening trick ( [6]; see also [9]). …”

Learning a Move-Generator for Upper Confidence Trees

“…What follows, is the formal description of the state of the art continuous MCTS, i.e. MCTS with Double Progressive Widening (MCTS-DPW), as seen in [3]. As the reader can see, it mainly requires two things: (i) a transition function, capable of simulating what happens when an action a is taken in state s, and returns a new state s ′ and a reward r. (ii) a default policy ϕ, capable of returning an action a, given a state s. When nothing is specified, it is assumed that this function returns a random action, following a random distribution that covers the entire set of feasible actions in state s. Let nbV isits(s) ← nbV isits(s) + 1 and let t = nbV isits(s) Let k = ⌈Ct α ⌉.…”

Consistent Belief State Estimation, with Application to Mines

“…n(s) is the number of simulations including state s, and n(s, a) is the number of simulations including action a in state s; they are all initialized to 0. Indeed, the implementation that we use (from the Mash project) is slightly more sophisticated and uses progressive widening [10], [11]; this is not relevant for this paper.…”