Robust control of the multi-armed bandit problem

Abstract

We study a robust model of the multi-armed bandit (MAB) problem in which the transition probabilities are ambiguous and belong to subsets of the probability simplex. We first show that for each arm there exists a robust counterpart of the Gittins index that is the solution to a robust optimal stopping-time problem and can be computed effectively with an equivalent restart problem. We then characterize the optimal policy of the robust MAB as a project-by-project retirement policy but we show that arms become dependent so the policy based on the robust Gittins index is not optimal. For a project selection problem, we show that the robust Gittins index policy is near optimal but its implementation requires more computational effort than solving a non-robust MAB problem. Hence, we propose a Lagrangian index policy that requires the same computational effort as evaluating the indices of a non-robust MAB and is within 1% of the optimum in the robust project selection problem.

Robust Control of Partially Observable Failing Systems

“…As pointed out by Chan andFarias (2009), Brown et al (2010), and Rogers (2007), even without the added difficulty of accounting for model miss-specification, solving dynamic optimization problems to optimality is difficult and one must often resort to heuristics. As such, other papers in this research area have diverted their efforts to developing methods for bounding the performance of heuristic policies relative to the robust optimal (e.g., Caro andGupta 2015, Kim andLim 2015). However, very few results regarding the structure of the optimal robust policy have been published in the robust dynamic optimization literature, which is the focus of this paper.…”

Optimal Learning Under Robustness and Time-Consistency

“…Thus (B µ t ) takes the same role as (W η t ) in CE (see (1) above). Rewrite(7) asdB µ t = − 1 σθ µ t (Z t ) dt +…”