A Tutorial on Thompson Sampling

Abstract

Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immediate performance and investing to accumulate new information that may improve future performance. The algorithm addresses a broad range of problems in a computationally efficient manner and is therefore enjoying wide use. This tutorial covers the algorithm and its application, illustrating concepts through a range of examples, including Bernoulli bandit problems, shortest path problems, product assortment, recommendation, active learning with neural networks, and reinforcement learning in Markov decision processes. Most of these problems involve complex information structures, where information revealed by taking an action informs beliefs about other actions. We will also discuss when and why Thompson sampling is or is not effective and relations to alternative algorithms.

On the Sample Complexity of the Linear Quadratic Regulator

“…In response to the computational intractability of the OFU principle, researchers in RL and online learning have proposed the use of Thompson sampling [49] for exploration. Abeille and Lazaric [2] show that the regret of a Thompson sampling approach for LQR scales as O(T 2/3 ) and improve the result to O( √ T ) in [3], where O(·) hides poly-logarithmic factors.…”

Switch-Based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning

“…Then, the active sampling routine (line 6 in Algorithm 1) can be expanded as Select the user goal i with the maximum p i value Here, N is the Gaussian distribution for introducing randomness. The Thompson-Sampling-like (Russo et al 2018) sub-routine of Algorithm 2 is motivated by two observations: (1) on average, categories with larger failure rate f i are more preferable as they inject more difficult cases (containing more useful information to be learned) based on the current performance of the agent policy. The generated data (simulated experiences) are generally associated with the steepest learning direction and can prospectively boost the training speed; (2) categories that are estimated less reliably (due to a smaller value of n i value) may have a large de facto failure rate, thus worth being allocated with more training instances to reduce the uncertainty.…”

Meta Dynamic Pricing: Learning Across Experiments

“…Following the literature on Thompson sampling, we consider a multivariate gaussian distribution since the posterior has a simple closed form, thereby admitting a tractable theoretical analysis. When implementing such an algorithm in practice, more complex distributions can be considered (e.g., see discussion in Russo et al 2018).…”

“…This prior captures shared structure of the kind we described above -e.g., the mean of the prior on the student-specific price-elasticity coefficient may be positive with a small standard deviation. It is well known that choosing a good (bad) prior significantly improves (hurts) the empirical performance of the algorithm (Chapelle and Li 2011, Honda and Takemura 2014, Liu and Li 2015, Russo et al 2018). However, the prior is typically unknown in practice, particularly when the decision-maker faces a cold start.…”