Linear Thompson sampling revisited

Abeille, Marc; Lazaric, Alessandro

doi:10.1214/17-ejs1341si

Cited by 134 publications

(203 citation statements)

References 15 publications

Supporting

Mentioning

196

Contrasting

Unclassified

Order By: Relevance

“…To mitigate this shortcoming, we propose a unified method based on the BLR approximation of these two methods. This unification is inspired by the analyses in Abeille and Lazaric [4], Abbasi-Yadkori et al [1] for linear bandits. 1) For LINPSRL: we deploy BLR to approximate the posterior distribution over the Q-function using conjugate Gaussian prior and likelihood.…”

Section: Strategymentioning

confidence: 98%

See 1 more Smart Citation

Efficient Exploration Through Bayesian Deep Q-Networks

Azizzadenesheli¹,

Brunskill

Anandkumar

2018

2018 Information Theory and Applications Workshop (ITA)

View full text Add to dashboard Cite

We study reinforcement learning (RL) in high dimensional episodic Markov decision processes (MDP). We consider value-based RL when the optimal Q-value is a linear function of d-dimensional state-action feature representation. For instance, in deep-Q networks (DQN), the Q-value is a linear function of the feature representation layer (output layer). We propose two algorithms, one based on optimism, LINUCB, and another based on posterior sampling, LINPSRL. We guarantee frequentist and Bayesian regret upper bounds of O(d √ T ) for these two algorithms, where T is the number of episodes. We extend these methods to deep RL and propose Bayesian deep Q-networks (BDQN), which uses an efficient Thompson sampling algorithm for high dimensional RL. We deploy the double DQN (DDQN) approach, and instead of learning the last layer of Q-network using linear regression, we use Bayesian linear regression, resulting in an approximated posterior over Q-function. This allows us to directly incorporate the uncertainty over the Q-function and deploy Thompson sampling on the learned posterior distribution resulting in efficient exploration/exploitation trade-off. We empirically study the behavior of BDQN on a wide range of Atari games. Since BDQN carries out more efficient exploration and exploitation, it is able to reach higher return substantially faster compared to DDQN.

show abstract

Section: Strategymentioning

confidence: 98%

“…1) For LINPSRL: we deploy BLR to approximate the posterior distribution over the Q-function using conjugate Gaussian prior and likelihood. In tabular MDP, this approach turns out to be similar to Osband Lazaric [4]). These two approximation procedures result in the same Gaussian distribution, and therefore, the same algorithm.…”

Section: Strategymentioning

confidence: 99%

Efficient Exploration Through Bayesian Deep Q-Networks

Azizzadenesheli¹,

Brunskill

Anandkumar

2018

2018 Information Theory and Applications Workshop (ITA)

View full text Add to dashboard Cite

show abstract

“…For MAB and LB, Agrawal and Goyal [2] proved a high probability regret bound. Abeille and Lazaric [19] showed the same regret bound in an alternative way and revealed conditions for variants of the TS algorithm to have such regret bounds. For the combinatorial semi-bandit problem and generalized problems including CLS, Wen et al [10] proved a regret bound regarding the Bayes cumulative regret proposed by Russo and Van Roy [23].…”

Section: Related Workmentioning

confidence: 75%

“…TS algorithms have been theoretically analyzed for several problems [2], [10], [16], [19]- [24]. For MAB and LB, Agrawal and Goyal [2] proved a high probability regret bound.…”

Section: Related Workmentioning

confidence: 99%

An Arm-Wise Randomization Approach to Combinatorial Linear Semi-Bandits

Takemura

Ito

2019

2019 IEEE International Conference on Data Mining (ICDM)

View full text Add to dashboard Cite

Combinatorial linear semi-bandits (CLS) are widely applicable frameworks of sequential decision-making, in which a learner chooses a subset of arms from a given set of arms associated with feature vectors. Existing algorithms work poorly for the clustered case, in which the feature vectors form several large clusters. This shortcoming is critical in practice because it can be found in many applications, including recommender systems. In this paper, we clarify why such a shortcoming occurs, and we introduce a key technique of arm-wise randomization to overcome it. We propose two algorithms with this technique: the perturbed C 2 UCB (PC 2 UCB) and the Thompson sampling (TS). Our empirical evaluation with artificial and real-world datasets demonstrates that the proposed algorithms with the armwise randomization technique outperform the existing algorithms without this technique, especially for the clustered case. Our contributions also include theoretical analyses that provide high probability asymptotic regret bounds for our algorithms.

show abstract

“…GT-TS workflow. GT-TS is a multi-armed bandit algorithm for experimental design that relies on two main concepts: i) an estimate of population's cell type discovery potential based on a variant of the Good-Toulmin estimator, and ii) a classic Thompson Sampling (TS) routine [Abeille andLazaric, 2017, Russo et al, 2017]. Pseudocode available in Supplementary Note 1.…”

Section: Discussionmentioning

confidence: 99%

GT-TS: Experimental design for maximizing cell type discovery in single-cell data

Dumitrascu

Feng

Engelhardt

2018

Preprint

View full text Add to dashboard Cite

We present the Good-Toulmin like estimator via Thompson sampling, a computational method for iterative experimental design in multi-tissue single-cell RNA-seq (scRNA-seq) data. Given a budget and modeling cell type information across tissues, GT-TS estimates how many cells are required for sampling from each tissue with the goal of maximizing cell type discovery across samples from multiple iterations. In both real and simulated data, we demonstrate the advantages of GT-TS in data collection planning when compared to a random strategy in the absence of experimental design.As both experimental and computational techniques advance, single-cell RNA-seq (scRNA-seq) allows for the characterization of cell type and cellular diversity at unprecedented high-throughout.Taking advantage of these developments, recent scientific efforts aim at the molecular profiling of all the cell types of an organism [Han et al., 2018, Regev et al., 2017. These initiatives raise important questions regarding experimental design choices. First, given a budget, how many cells

show abstract

Linear Thompson sampling revisited

Cited by 134 publications

References 15 publications

Efficient Exploration Through Bayesian Deep Q-Networks

Efficient Exploration Through Bayesian Deep Q-Networks

An Arm-Wise Randomization Approach to Combinatorial Linear Semi-Bandits

GT-TS: Experimental design for maximizing cell type discovery in single-cell data

Contact Info

Product

Resources

About