Robust Control of the Multi-Armed Bandit Problem

Caro, Felipe; Gupta, Aparupa Das

doi:10.2139/ssrn.2326583

Cited by 6 publications

(19 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The study of an adversary for the payoff in the bandit problem (via Gittins index) has been considered by Caro and Gupta [20] and Kim and Lim [51] (with additional penalty in the reward) using Whittle's retirement option argument [76]. In their works, they rely heavily on a Markov assumption, which allows them to postulate a robust dynamic programming principle (see also Iyengar [46], Nilim and El Ghaoui [58]).…”

Section: Multi-armed Banditsmentioning

confidence: 99%

“…For example, statistical concerns are treated in this framework in [23,24] or Bielecki, Cialenco and Chen [14]. We could also allow adversarial choices with a range of a fixed set (as in the classical adversarial bandit problem [7] or as in [20,51]) or a random set which can be used to model learning as in the classical Gittins' theory. We also allow dynamic adversaries, which are not considered in the usual adversarial setting.…”

Section: Multi-armed Banditsmentioning

confidence: 99%

See 1 more Smart Citation

Gittins’ theorem under uncertainty

Cohen

Treetanthiploet

2022

Electron. J. Probab.

View full text Add to dashboard Cite

We study dynamic allocation problems for discrete time multi-armed bandits under uncertainty, based on the the theory of nonlinear expectations. We show that, under independence assumption on the bandits and with some relaxation in the definition of optimality, a Gittins allocation index gives optimal choices. This involves studying the interaction of our uncertainty with controls which determine the filtration. We also run a simple numerical example which illustrates the interaction between the willingness to explore and uncertainty aversion of the agent when making decisions.

show abstract

Section: Multi-armed Banditsmentioning

confidence: 99%

Section: Multi-armed Banditsmentioning

confidence: 99%

Gittins’ theorem under uncertainty

Cohen

Treetanthiploet

2022

Electron. J. Probab.

View full text Add to dashboard Cite

show abstract

“…We know of one other paper on robust bandit problems, which is the recent paper by Caro and Gupta (2015). We briefly highlight the differences.…”

Section: Relevant Literaturementioning

confidence: 99%

“…We briefly highlight the differences. First, our paper adopts a different approach for expressing model uncertainty; Caro and Gupta (2015) adopts the constraint approach whereas we adopt the penalty approach. The differences between these approaches are discussed in §3.1.…”

Section: Relevant Literaturementioning

confidence: 99%

“…The differences between these approaches are discussed in §3.1. Second, Caro and Gupta (2015) focus on numerical algorithms for computing proposed index policies in the constraint setting. On the other hand, we spend little time on this issue because the policy we propose can be computed by using obvious extensions of algorithms for a classical bandit problem.…”

Section: Relevant Literaturementioning

confidence: 99%

See 1 more Smart Citation

Robust Multiarmed Bandit Problems

Kim

Lim

2016

Management Science

View full text Add to dashboard Cite

T he multiarmed bandit problem is a popular framework for studying the exploration versus exploitation trade-off. Recent applications include dynamic assortment design, Internet advertising, dynamic pricing, and the control of queues. The standard mathematical formulation for a bandit problem makes the strong assumption that the decision maker has a full characterization of the joint distribution of the rewards, and that "arms" under this distribution are independent. These assumptions are not satisfied in many applications, and the outof-sample performance of policies that optimize a misspecified model can be poor. Motivated by these concerns, we formulate a robust bandit problem in which a decision maker accounts for distrust in the nominal model by solving a worst-case problem against an adversary ("nature") who has the ability to alter the underlying reward distribution and does so to minimize the decision maker's expected total profit. Structural properties of the optimal worst-case policy are characterized by using the robust Bellman (dynamic programming) equation, and arms are shown to be no longer independent under nature's worst-case response. One implication of this is that index policies are not optimal for the robust problem, and we propose, as an alternative, a robust version of the Gittins index. Performance bounds for the robust Gittins index are derived by using structural properties of the value function together with ideas from stochastic dynamic programming duality. We also investigate the performance of the robust Gittins index policy when applied to a Bayesian webpage design problem. In the presence of model misspecification, numerical experiments show that the robust Gittins index policy not only outperforms the classical Gittins index policy, but also substantially reduces the variability in the out-of-sample performance.

show abstract

Robust experimentation in the continuous time bandit problem

Pourbabaee

2020

Econ Theory

View full text Add to dashboard Cite

We study the experimentation dynamics of a decision maker (DM) in a two-armed bandit setup (Bolton and Harris (1999)), where the agent holds ambiguous beliefs regarding the distribution of the return process of one arm and is certain about the other one. The DM entertains Multiplier preferences à la Hansen and Sargent ( 2001), thus we frame the decision making environment as a two-player differential game against nature in continuous time. We characterize the DM's value function and her optimal experimentation strategy that turns out to follow a cut-off rule with respect to her belief process. The belief threshold for exploring the ambiguous arm is found in closed form and is shown to be increasing with respect to the ambiguity aversion index. We then study the effect of provision of an unambiguous information source about the ambiguous arm. Interestingly, we show that the exploration threshold rises unambiguously as a result of this new information source, thereby leading to more conservatism. This analysis also sheds light on the efficient time to reach for an expert opinion.

show abstract

Robust Control of the Multi-Armed Bandit Problem

Cited by 6 publications

References 22 publications

Gittins’ theorem under uncertainty

Gittins’ theorem under uncertainty

Robust Multiarmed Bandit Problems

Robust experimentation in the continuous time bandit problem

Contact Info

Product

Resources

About