Robust Multiarmed Bandit Problems

Abstract

T he multiarmed bandit problem is a popular framework for studying the exploration versus exploitation trade-off. Recent applications include dynamic assortment design, Internet advertising, dynamic pricing, and the control of queues. The standard mathematical formulation for a bandit problem makes the strong assumption that the decision maker has a full characterization of the joint distribution of the rewards, and that "arms" under this distribution are independent. These assumptions are not satisfied in many applications, and the outof-sample performance of policies that optimize a misspecified model can be poor. Motivated by these concerns, we formulate a robust bandit problem in which a decision maker accounts for distrust in the nominal model by solving a worst-case problem against an adversary ("nature") who has the ability to alter the underlying reward distribution and does so to minimize the decision maker's expected total profit. Structural properties of the optimal worst-case policy are characterized by using the robust Bellman (dynamic programming) equation, and arms are shown to be no longer independent under nature's worst-case response. One implication of this is that index policies are not optimal for the robust problem, and we propose, as an alternative, a robust version of the Gittins index. Performance bounds for the robust Gittins index are derived by using structural properties of the value function together with ideas from stochastic dynamic programming duality. We also investigate the performance of the robust Gittins index policy when applied to a Bayesian webpage design problem. In the presence of model misspecification, numerical experiments show that the robust Gittins index policy not only outperforms the classical Gittins index policy, but also substantially reduces the variability in the out-of-sample performance.

Robust Control of Partially Observable Failing Systems

“…In this regard, one can view P and T as a tuning parameters that parametrizes a family of policies of increasing robustness with decreasing P and T . In practice, to select specific values of the ambiguity parameters using historical data, the parameters can be tuned using outof-sample tests (see, for example, the recent papers of Lim 2015 andGotoh et al 2016 who consider this issue in detail and propose a number of robust cross-validation procedures to select ambiguity parameters from data).…”

“…In the constraint approach, the set of alternative models is represented as a hard constraint, and confidence in the nominal is captured by the size of this uncertainty set (see, e.g., Ben-Tal and Nemirovski 1998, 1999, 2000Bertsimas and Sim 2004;El Ghaoui and Lebret 1997;Iyengar 2005;Li and Kwon 2013;Nilim and El Ghaoui 2005;Wiesemann et al 2013). The penalty approach on the other hand expresses confidence in the nominal by penalizing alternative models that deviate too far from the nominal, and does so via a penalty function (soft constraint) that appears in the objective function (see, e.g., Dai Pra et al 1996, Peterson et al 2000, Hansen and Sargent 2007, Jain et al 2010, Kim and Lim 2015, Lim and Shanthikumar 2007. In this paper, we adopt an entropy penalty approach to represent model missspecification, because it provides a number of advantages from the perspective of characterizing the structure of the optimal robust control policy.…”

“…As pointed out by Chan andFarias (2009), Brown et al (2010), and Rogers (2007), even without the added difficulty of accounting for model miss-specification, solving dynamic optimization problems to optimality is difficult and one must often resort to heuristics. As such, other papers in this research area have diverted their efforts to developing methods for bounding the performance of heuristic policies relative to the robust optimal (e.g., Caro andGupta 2015, Kim andLim 2015). However, very few results regarding the structure of the optimal robust policy have been published in the robust dynamic optimization literature, which is the focus of this paper.…”

A Bayesian two‐armed bandit model

“…With this approach, a bandit process is naturally formulated as a Markov decision process, the solution of which is often obtained through stochastic dynamic programming. In addition to the aforementioned applications in business and clinical trials, the bandit model is frequently used as a theoretical framework for many other research problems, including stochastic scheduling, queueing networks, optimal investment and consumption, dynamic assortment design, modern online service, and webpage design …”

Robust control of the multi-armed bandit problem

“…This approach has been extended to finite state and action MDPs with incomplete information (Burnetas and Katehakis [6]) and to adversarial bandits that either make no assumption whatsoever on the process generating the payoffs of the bandits (Auer et al [1]) or bound its variation within a "variation budget" (Besbes et al [4]). At the time of submission we became aware of the work by Kim and Lim [18] that also study the RMAB problem but with an alternative formulation in which deviations of the transition probabilities from their point estimates are penalized, so the analysis is essentially different from ours.…”