Optimal Policy for Bernoulli Bandits: Computation and Algorithm Gauge

Abstract

Bernoulli multi-armed bandits are a reinforcement learning model used to study a variety of choice optimization problems. Often such optimizations concern a finite-time horizon. In principle, statistically optimal policies can be computed via dynamic programming, but doing so is considered infeasible due to prohibitive computational requirements and implementation complexity. Hence, suboptimal algorithms are applied in practice, despite their unknown level of suboptimality. In this article, we demonstrate that optimal policies can be efficiently computed for large time horizons or number of arms thanks to a novel memory organization and indexing scheme. We use optimal policies to gauge the suboptimality of several well-known finite-and infinitetime horizon algorithms including Whittle and Gittins indices, epsilon-greedy, Thompson sampling, and upper-confidence bound (UCB) algorithms. Our simulation study shows that all but one evaluated algorithm perform significantly worse than the optimal policy. The Whittle index offers a nearly optimal strategy for multiarmed Bernoulli bandits despite its suboptimal decisions-up to 10%-compared to an optimal policy table. Lastly, we discuss optimizations of known algorithms. We derive a novel solution from UCB1-tuned. It outperforms other infinite-time horizon algorithms when dealing with many arms.Impact statement-Bernoulli bandits are a reinforcement learning model used to improve decisions with binary outcomes. They have various applications ranging from headline news selection to clinical trials. Existing bandit algorithms are suboptimal. This article provides the first practical computation method, which determines the optimal decisions in Bernoulli bandits. It provides the lowest achievable decision regret (maximum expected benefit). In clinical trials, where an algorithm selects treatments for subsequent patients, our method can substantially reduce the number of unsuccessfully treated patients-by up to 5×. The optimal strategy is also used for new comprehensive evaluations of well-known suboptimal algorithms. This can significantly improve decision effectiveness in various applications. Index Terms-Clinical trials, epsilon-greedy, Gittins index (GI), multi-armed Bernoulli bandits, optimal policy (OPT), POKER, Thompson sampling (TS), upper-confidence bound (UCB), Whittle index (WI).

Actively Tracking the Optimal Arm in Non-Stationary Environments with Mandatory Probing

“…• We propose a coded grouping of the arms based on the arm indices and consequently, derive the probability of missed detection of change and the probability of false alarm and highlight the conditions to limit these probabilities. Based on this, we show that TS-GE achieves sub-linear regret, 5 } . We compare this bound with the best known bound of O( √ KN C T log T ) and discuss the conditions under which the bound of TS-GE outperforms the latter.…”

“…Sequential decision making problems in reinforcementlearning (RL) are popularly formulated using the multi-armed bandit (MAB) framework, wherein, an agent (or player) selects one or multiple options (or arms) out of a set of arms at each time slot [1]- [5]. The player performs such an action-selection based on the current estimate or belief of the expected reward of the arms.…”

Delayed Reward Bernoulli Bandits: Optimal Policy and Predictive Meta-Algorithm PARDI

“…There exists a wealth of published work on multi-armed bandits under a variety of assumptions [8], [14], [15], [18]- [20] with some work focusing exclusively on Bernoulli bandits [2]- [4], [6], [16], [21]- [23]. However, as stressed in [14], few articles have been published on any variation of bandits with delayed rewards.…”

“…Problem statement: Practically feasible methods to compute the optimal policy in the context of Bernoulli bandits for immediate rewards were recently proposed in [16], where the level of suboptimality of well-known algorithms was gauged. However, to the best of our knowledge, no computation of the optimal policy under delays has ever been published or proposed.…”

“…They differ in assumptions and performances [8], [14], [15]. While some algorithms are general, others were designed specifically for Bernoulli bandits [2], [4], [16]. Some algorithms assume that the game goes forever.…”