Simple Bayesian Algorithms for Best-Arm Identification

Abstract

This paper considers the optimal adaptive allocation of measurement effort for identifying the best among a finite set of options or designs. An experimenter sequentially chooses designs to measure and observes noisy signals of their quality with the goal of confidently identifying the best design after a small number of measurements. This paper proposes three simple and intuitive Bayesian algorithms for adaptively allocating measurement effort, and formalizes a sense in which these seemingly naive rules are the best possible. One proposal is top-two probability sampling, which computes the two designs with the highest posterior probability of being optimal, and then randomizes to select among these two. One is a variant of top-two sampling which considers not only the probability a design is optimal, but the expected amount by which its quality exceeds that of other designs. The final algorithm is a modified version of Thompson sampling that is tailored for identifying the best design.We prove that these simple algorithms satisfy a sharp optimality property. In a frequentist setting where the true quality of the designs is fixed, one hopes the posterior definitively identifies the optimal design, in the sense that that the posterior probability assigned to the event that some other design is optimal converges to zero as measurements are collected. We show that under the proposed algorithms this convergence occurs at an exponential rate, and the corresponding exponent is the best possible among all allocation rules. It should be highlighted that the proposed algorithms depend on a single tuning parameter, which determines the probability used when randomizing among the top-two designs. Attaining the optimal rate of posterior convergence requires either that this parameter is set optimally or is tuned adaptively toward the optimal value. The paper goes further, characterizing the exponent attained on any problem instance and for any value of the tunable parameter. This exponent is interpreted as being optimal among a constrained class of allocation rules. Finally, considerable robustness to this parameter is established through numerical experiments and theoretical results. When this parameter is set to 1/2, the exponent attained is within a factor of 2 of best possible across all problem instances.

Shrinkage Estimators in Online Experiments

“…In essence, then, Thompson sampling with empirical Bayes shrinkage behaves similarly to algorithms like "top two Thompson sampling" (Russo, 2016). In this algorithm, Thompson sampling incorporates information about both of the top two performing arms in its construction of a posterior of the probability of an arm lying in the top two ranks.…”

Information theory for ranking and selection

“…Many algorithms have been developed for R&S, including procedural approaches (Rinott, 1978), algorithms meant for solving large-scale R&S problems using fully sequential procedures (Luo, Hong, Nelson, & Wu, 2015), optimal sampling algorithm (Hunter & Pasupathy, 2013), optimal budget allocation methods (Chen, Donohue, Yücesan, & Lin, 2003), and so on. In this study, we use six different algorithms as benchmarks to compare against our proposed policy; namely, knowledge gradient (KG) by Frazier, Powell, and Dayanik (2008), top-two Thompson sampling (TTTS) by Russo (2016) which promotes more exploration compared to Thompson (posterior) sampling (TS), the most starving sequential optimal computing budget allocation (OCBA) by Chen and Lee (2011) which is modified for the Bayesian setting, general Bayesian budget allocation (GBBA) by Peng, Chen, Fu, and Hu (2016) which uses a randomized allocation function for OCBA, and approximately optimal allocation policy (AOAP) by Peng, Chong, Chen, and Fu (2018) which allocates samples in a myopic sense with the goal of minimizing the largest coefficient of variation of the difference between the best alternative (with the largest posterior mean) and the rest.…”

Bayesian Best-Arm Identification for Selecting Influenza Mitigation Strategies

“…Additionally, at the time an arm is recommended, the posteriors contain valuable information that can be used to formulate a variety of statistics helpful to assist decision makers. We consider two state-of-the-art Bayesian algorithms: BayesGap [33] and Top-two Thompson sampling [51]. For Top-two Thompson sampling, we derive a statistic based on the posteriors to inform the decision makers about the confidence of an arm recommendation: the probability of success.…”