The Finite-Horizon Two-Armed Bandit Problem with Binary Responses: A Multidisciplinary Survey of the History, State of the Art, and Myths

Jacko, Peter

doi:10.48550/arxiv.1906.10173

Cited by 3 publications

(3 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the interested reader, Rosenberger and Lachin (2016, Section 10.2) gives a brief summary of the history of both of these areas, and an overview of multi-arm bandit models is presented in the review paper of Villar et al (2015a). For a review of non-randomized algorithms for the two-arm bandit problem, see Jacko (2019).…”

Section: Rar Methodology Literaturementioning

confidence: 99%

Response-adaptive randomization in clinical trials: from myths to practical considerations

Robertson¹,

Lee²,

Lopez-Kolkovska³

et al. 2020

Preprint

View full text Add to dashboard Cite

Response-adaptive randomization (RAR) is part of a wider class of data-dependent sampling algorithms, for which clinical trials have commonly been used as a motivating application. In that context, patient allocation to treatments is defined using the accrued data on responses to alter randomization probabilities, in order to achieve different experimental goals. RAR has received abundant theoretical attention from the biostatistical literature since the 1930's and has been the subject of heated debates. Recently it has received renewed consideration from the applied community due to successful practical examples and its widespread use in machine learning. Many position papers on the subject present a one-sided view on its use, which is of limited value for the non-expert. This work aims to address this persistent gap by providing a critical, balanced and updated review of methodological and practical issues to consider when debating the use of RAR in clinical trials.

show abstract

Section: Rar Methodology Literaturementioning

confidence: 99%

Response-adaptive randomization in clinical trials: from myths to practical considerations

Robertson¹,

Lee²,

Lopez-Kolkovska³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…There exists a wealth of published work on multi-armed bandits under a variety of assumptions [8], [14], [15], [18]- [20] with some work focusing exclusively on Bernoulli bandits [2]- [4], [6], [16], [21]- [23]. However, as stressed in [14], few articles have been published on any variation of bandits with delayed rewards.…”

Section: A Related Work and Motivationmentioning

confidence: 99%

Delayed Reward Bernoulli Bandits: Optimal Policy and Predictive Meta-Algorithm PARDI

Pilarski

Pilarski²,

Varró

2022

IEEE Trans. Artif. Intell.

View full text Add to dashboard Cite

Bernoulli multi-armed bandits are a reinforcement learning model used to optimize the sequences of decisions with binary outcomes. Well-known bandit algorithms, including the optimal policy, assume that before a decision is made the outcomes of previous decisions are known. This assumption is often not satisfied in real-life scenarios. As demonstrated in this article, if decision outcomes are affected by delays, the performance of existing algorithms can be severely affected. We present the first practically applicable method to compute statistically optimal decisions in the presence of outcome delays. Our method has a predictive component abstracted out into a meta-algorithm, predictive algorithm reducing delay impact (PARDI), which significantly reduces the impact of delays on commonly used algorithms. We demonstrate empirically that PARDI-enhanced Whittle index is nearly optimal for a wide range of Bernoulli bandit parameters and delays. In a wide spectrum of experiments, it performed better than any other suboptimal algorithm, e.g., UCB1-tuned and Thompson sampling. PARDI-enhanced Whittle index can be used when computational requirements of the optimal policy are too high.Impact Statement-Bernoulli multi-armed bandit algorithms are used to optimize sequential binary decisions. Oftentimes, decisions must be made without knowing the results of some previous decisions, e.g., in clinical trials where finding out treatment outcomes takes time. Well-known bandit algorithms are ill-equipped to deal with still unknown (delayed) decision results, which may translate into significant losses, e.g., the number of unsuccessfully treated patients. We present the first method of determining the optimal strategy for these type of situations and a meta-algorithm PARDI that drastically improves the quality of decisions by wellknown algorithms-lowers regret by up to 3×. This is achieved by a 6× reduction in excess regret caused by delay. By addressing delays, this work can improve the quality of decisions in various applications. It opens new applications of Bernoulli bandits.

show abstract

“…This paper will use the clinical trial setting as our main focus. Jacko [30] illustrates the formulated terminology of MABPs in various disciplines, and we follow the terminology that is widely used in the machine learning and clinical trial communities. For instance, 'algorithm' and 'arms' in MABPs correspond to 'design' and 'treatments' in biostatistics (see Table 1 in [30]).…”

Section: Introductionmentioning

confidence: 99%

Some performance considerations when using multi-armed bandit algorithms in the presence of missing data

Chen,

Lee,

Villar

et al. 2022

Preprint

View full text Add to dashboard Cite

When using multi-armed bandit algorithms, the potential impact of missing data is often overlooked. In practice, the simplest approach is to ignore missing outcomes and continue to sample following the bandit algorithm. We investigate the impact of missing data on several bandit algorithms via a simulation study assuming the rewards are missing at random. We focus on two-armed bandit algorithms with binary outcomes in the context of patient allocation for clinical trials with relatively small sample sizes. However, our results can apply to other applications of bandit algorithms where missing data is expected to occur. We assess the resulting operating characteristics, including the expected reward (i.e., allocation results). Different probabilities of missingness in both arms are considered. The key finding of our work is that when using the simplest strategy of ignoring missing data, the corresponding impact on the performance of multi-armed bandit strategies varies according to their way of balancing the exploration-exploitation trade-off. Algorithms that are geared towards exploration continue to assign samples to the arm with more missing responses, and this arm is perceived as the superior arm by the algorithm. By contrast, algorithms that are geared towards exploitation would do the opposite and not assign samples to the arms with more missing responses. Furthermore, for algorithms focusing more on exploration, we illustrate that the problem of missing responses can be alleviated using a simple mean imputation approach.

show abstract

The Finite-Horizon Two-Armed Bandit Problem with Binary Responses: A Multidisciplinary Survey of the History, State of the Art, and Myths

Cited by 3 publications

References 45 publications

Response-adaptive randomization in clinical trials: from myths to practical considerations

Response-adaptive randomization in clinical trials: from myths to practical considerations

Delayed Reward Bernoulli Bandits: Optimal Policy and Predictive Meta-Algorithm PARDI

Some performance considerations when using multi-armed bandit algorithms in the presence of missing data

Contact Info

Product

Resources

About