Restless bandits, partial conservation laws and indexability

Abstract

We show that if performance measures in a stochastic scheduling problem satisfy a set of so-called partial conservation laws (PCL), which extend previously studied generalized conservation laws (GCL), then the problem is solved optimally by a priority-index policy for an appropriate range of linear performance objectives, where the optimal indices are computed by a one-pass adaptive-greedy algorithm, based on Klimov's. We further apply this framework to investigate the indexability property of restless bandits introduced by Whittle, obtaining the following results: (1) we identify a class of restless bandits (PCL-indexable) which are indexable; membership in this class is tested through a single run of the adaptive-greedy algorithm, which also computes the Whittle indices when the test is positive; this provides a tractable sufficient condition for indexability; (2) we further identify the class of GCL-indexable bandits, which includes classical bandits, having the property that they are indexable under any linear reward objective. The analysis is based on the so-called achievable region method, as the results follow from new linear programming formulations for the problems investigated.

Admission Control and Routing to Parallel Queues with Delayed Information via Marginal Productivity Indices

“…Instead of proposing some ad hoc index via heuristic arguments, we will deploy a unifying fundamental design principle for priority allocation policies in multiarmed restless bandit problems (MARBPs), of which (2) and (3) are special cases, based on the economically intuitive concept of marginal productivity index (MPI). Such an approach was introduced in [15], and has been developed and applied in a variety of models by the second author in work including [7,8,10,9], which is reviewed in [12]. In particular, [8,13] introduced such an approach to the design of index policies for admission control and routing to parallel exponential queues without delayed information.…”

“…Two issues need thus be addressed: (i) show that problem (4) is indeed indexable; and (ii) design an efficient indexcomputing algorithm. As for the first issue, we will deploy the sufficient indexability conditions based on partial conservation laws (PCLs) introduced in [7,8]. Such conditions require one to identify a family of stationary deterministic policies among which an optimal policy for problem (4) exists for every value of the parameter ν.…”

“…As for the second issue, that of index computation, provided PCL-indexability is established relative to such a family of policies, one can use the adaptive-greedy index algorithm introduced in [7,8] to compute the MPI. Using the general fast-pivoting implementation given in [11] such an algorithm has a cubic arithmetic operation complexity in the number of restless bandit states, which in the present setting corresponds to an O(n 3 ) operation count.…”

Stochastic dynamic programming based approaches to sensor resource management

“…Although there has been much research on index rules [7], 161, [3], [9], [5], there appear to be no applications to S R M except for the recent work by Krishnamurthy and Evans [4]. The reason for this is that the SRM problem, formulated as stochastic scheduling, doesn't usually satisfy the conditions necessary for the existence of an index rule solution.…”

“…The reason for this is that the SRM problem, formulated as stochastic scheduling, doesn't usually satisfy the conditions necessary for the existence of an index rule solution. Thus, one is forced to develop approximations (e.g., the "restless bandit" work of [9], [5]). In this paper we show how to relax one of the "usual" conditions and to develop index rule-based SRM algorithms for kinematic tracking.…”