Kullback–Leibler upper confidence bounds for optimal sequential allocation

Cappé, Olivier; Garivier, Aurélien; Maillard, Odalric-Ambrym; Munos, Rémi; Stoltz, Gilles

doi:10.1214/13-aos1119

Cited by 256 publications

(316 citation statements)

References 30 publications

Supporting

Mentioning

309

Contrasting

Unclassified

Order By: Relevance

“…Notice that [81] and [37] also provided an algorithm with asymptotic guarantees (under more restrictive conditions). It is only in [54,85,39] that a finite-time analysis was derived for KL-based UCB algorithms, KL-UCB and K inf -UCB, that achieve the asymptotic lower bounds of [81] and [37] respectively. Those algorithms make use of KL divergences in the definition of the UCBs and use the full empirical reward distribution (and not only the two first moments).…”

Section: Recent Improvementsmentioning

confidence: 99%

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Munos

2014

FNT in Machine Learning

197

181

View full text Add to dashboard Cite

This work covers several aspects of the optimism in the face of uncertainty principle applied to large scale optimization problems under finite numerical budget. The initial motivation for the research reported here originated from the empirical success of the so-called Monte-Carlo Tree Search method popularized in Computer Go and further extended to many other games as well as optimization and planning problems. Our objective is to contribute to the development of theoretical foundations of the field by characterizing the complexity of the underlying optimization problems and designing efficient algorithms with performance guarantees. The main idea presented here is that it is possible to decompose a complex decision making problem (such as an optimization problem in a large search space) into a sequence of elementary decisions, where each decision of the sequence is solved using a (stochastic) multi-armed bandit (simple mathematical model for decision making in stochastic environments). This so-called hierarchical bandit approach (where the reward observed by a bandit in the hierarchy is itself the return of another bandit at a deeper level) possesses the nice feature of starting the exploration by a quasi-uniform sampling of the space and then focusing progressively on the most promising area, at different scales, according to the evaluations observed so far, and eventually performing a local search around the global optima of the function. The performance of the method is assessed in terms of the optimality of the returned solution as a function of the number of function evaluations. Our main contribution to the field of function optimization is a class of hierarchical optimistic algorithms designed for general search spaces (such as metric spaces, trees, graphs, Euclidean spaces, ...) with different algorithmic instantiations depending on whether the evaluations are noisy or noiseless and whether some measure of the "smoothness" of the function is known or unknown. The performance of the algorithms depends on the local behavior of the function around its global optima expressed in terms of the quantity of near-optimal states measured with some metric. If this local smoothness of the function is known then one can design very efficient optimization algorithms (with convergence rate independent of the space dimension), and when it is not known, we can build adaptive techniques that can, in some cases, perform almost as well as when it is known. In order to be self-contained, we start with a brief introduction to the stochastic multi-armed bandit problem in Chapter 1 and describe the UCB (Upper Confidence Bound) strategy and several extensions. In Chapter 2 we present the Monte-Carlo Tree Search method applied to Computer Go and show the limitations of previous algorithms such as UCT (UCB applied to Trees). This provides motivation for designing theoretically well-founded optimistic optimization algorithms. The main contributions on hierarchical optimistic optimization are described in Chapters 3 and 4...

show abstract

Section: Recent Improvementsmentioning

confidence: 99%

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Munos

2014

FNT in Machine Learning

197

181

View full text Add to dashboard Cite

show abstract

“…In particular, the work of (May and Leslie 2011) and the work of (Granmo 2010) prove asymptotic convergence of Thompson sampling. The performance of bandit algorithms has also been studied in terms of the rate of growth of the regret (Lai and Robbins 1995), and recent bandit algorithms have been shown to match this lower bound (Cappé et al 2013), including Thompson sampling algorithms for Bernoulli bandits (Kaufmann et al 2012). Also, the work of (Chapelle and Li 2011) presents empirical results that show Thompson sampling is highly competitive, matching or outperforming popular methods such as UCB (Lai and Robbins 1995;Auer et al 2002).…”

Section: Discussionmentioning

confidence: 99%

Generalized Thompson sampling for sequential decision-making and causal inference

Ortega

Braun

2014

Complex Adapt Syst Model

View full text Add to dashboard Cite

Purpose: Sampling an action according to the probability that the action is believed to be the optimal one is sometimes called Thompson sampling. Methods: Although mostly applied to bandit problems, Thompson sampling can also be used to solve sequential adaptive control problems, when the optimal policy is known for each possible environment. The predictive distribution over actions can then be constructed by a Bayesian superposition of the policies weighted by their posterior probability of being optimal. Results: Here we discuss two important features of this approach. First, we show in how far such generalized Thompson sampling can be regarded as an optimal strategy under limited information processing capabilities that constrain the sampling complexity of the decision-making process. Second, we show how such Thompson sampling can be extended to solve causal inference problems when interacting with an environment in a sequential fashion. Conclusion: In summary, our results suggest that Thompson sampling might not merely be a useful heuristic, but a principled method to address problems of adaptive sequential decision-making and causal inference.

show abstract

“…A good framework we use is called pymaBandits [15]. It already includes several learning policies like the Gittin's index, the classical UCB policy and some variations of it, the MOSS policy and some others.…”

Section: Methodsmentioning

confidence: 99%

“…The extension to multiple Bernoulli distributed experiments (MAB problem) corresponds to a situation where each move can be successful, but only one move has the highest success probability to win. An efficient implementation of the Bernoulli multi-armed bandit problem and several learning algorithms was done by Olivier Cappé et al [15]. We use this implementation to conduct our experiments generating the decision policies and reward processes.…”

Section: Multi-armed Banditsmentioning

confidence: 99%

Emotional evaluation of bandit problems

Feldmaier

Diepold

2013

2013 Ieee Ro-Man

View full text Add to dashboard Cite

Abstract-In this paper, we discuss an approach to evaluate decisions made during a multi-armed bandit learning experiment. Usually, the results of machine learning algorithms applied on multi-armed bandit scenarios are rated in terms of earned reward and optimal decisions taken. These criteria are valuable for objective comparison in finite experiments. But learning algorithms used in real scenarios, for example in robotics, need to have instantaneous criteria to evaluate their actual decisions taken.[1] To overcome this problem, in our approach each decision updates the Zürich model which emulates the human sense of feeling secure and aroused. Combining these two feelings results in an emotional evaluation of decision policies and could be used to model the emotional state of an intelligent agent.

show abstract

Kullback–Leibler upper confidence bounds for optimal sequential allocation

Cited by 256 publications

References 30 publications

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning

Generalized Thompson sampling for sequential decision-making and causal inference

Emotional evaluation of bandit problems

Contact Info

Product

Resources

About