Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Abstract

We consider the multi-armed bandit problem with penalties for switching that include setup delays and costs, extending the former results of the author for the special case with no switching delays. A priority index for projects with setup delays that characterizes, in part, optimal policies was introduced by Asawa and Teneketzis in 1996, yet without giving a means of computing it. We present a fast two-stage index computing method, which computes the continuation index (which applies when the project has been set up) in a first stage and certain extra quantities with cubic (arithmetic-operation) complexity in the number of project states and then computes the switching index (which applies when the project is not set up), in a second stage, with quadratic complexity. The approach is based on new methodological advances on restless bandit indexation, which are introduced and deployed herein, being motivated by the limitations of previous results, exploiting the fact that the aforementioned index is the Whittle index of the project in its restless reformulation. A numerical study demonstrates substantial runtime speed-ups of the new two-stage index algorithm versus a general one-stage Whittle index algorithm. The study further gives evidence that, in a multi-project setting, the index policy is consistently nearly optimal.

Multi-Gear Bandits, Partial Conservation Laws, and Indexability

“…The latter has huge modeling power but is computationally intractable, and Whittle's index policy has proven effective in an ever-increasing variety of models for multifarious applications. Thus, e.g., to name a few, scheduling multi-class make-to-stock queues [9], scheduling multi-class queues with finite buffers [10], admission control and routing to parallel queues with reneging [11], obsolescence mitigation strategies [12], sensor scheduling and dynamic channel selection [13][14][15][16], group maintenance [17], multi-target tracking with Kalman filter dynamics [18,19], scheduling multi-armed bandits with switching costs [20] or switching delays [21], the dynamic prioritization of medical treatments or interventions [22,23], and resource allocation with varying requests and with resources shared by multiple requests [24].…”