A Verification Theorem for Threshold-Indexability of Real-State Discounted Restless Bandits

Abstract

The Whittle index, which characterizes optimal policies for controlling certain single restless bandit projects (a Markov decision process with two actions: active and passive) is the basis for a widely used heuristic index policy for the intractable restless multiarmed bandit problem. Yet two roadblocks need to be overcome to apply such a policy: the individual projects in the model at hand must be shown to be indexable, so that they possess a Whittle index; and the index must be evaluated. Such roadblocks can be especially vexing when project state spaces are real intervals, as in recent sensor scheduling applications. This paper presents sufficient conditions for indexability (relative to a generalized Whittle index) of general real-state discrete-time restless bandits under the discounted criterion, which are not based on elucidating properties of the optimal value function and do not require proving beforehand optimality of threshold policies as in prevailing approaches. The main contribution is a verification theorem establishing that, if project performance metrics under threshold policies and an explicitly defined marginal productivity (MP) index satisfy three conditions, then the project is indexable with its generalized Whittle index being given by the MP index, and threshold policies are optimal for dynamic project control.

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

“…We next aim to establish that condition (ii) in Theorem 1(a) is satisfied by the present model, i.e., that F -policies, i.e., those with active sets S 0 ⊕ S 1 ∈ F that are defined by (35), suffice to solve the λ-price problem (33) for any price λ ∈ R. We will use the DP optimality equations that characterize the optimal value function V * (a − ,i) (λ) for problem (33), starting from each augmented state (a − , i) ∈ Y: thus, for each original state i ∈ X ,…”

“…In that way, the MABP with switching costs is cast as a multi-armed restless bandit problem (MARBP) without them, which allows for the deployment of theoretical and algorithmic results on restless bandit indexation, as introduced in [29] by Whittle. Such a theory has been developed in [30][31][32][33] by the author. Additionally, see the survey [34].…”

“…Proof. We start by showing that λ * (1,i) = λ AT (1,i) , while using the equivalences λ λ * (1,i) ⇐⇒ resting the project in (1, i) is optimal for problem (33) ⇐⇒ 0 max…”

A Fast-Pivoting Algorithm for Whittle’s Restless Bandit Index

“…Under indexability, the P λ -optimal active and passive sets S * ,1 (λ) and S * ,0 (λ) are characterized by an index attached to project states. Note that the definition below refers to the sign function sgn : R → {−1, 0, 1}, and follows the formulation of indexability in ( [25], Definition 1). Definition 1 (Indexability and Whittle index).…”

“…Typically, researchers use ad hoc analyses to prove indexability and calculate the Whittle index in particularly models. In contrast, the author has introduced and developed in [3,24] a methodology to establish indexability and compute the Whittle index for general finite-state restless bandits, extended to the semi-Markov denumerable-state case in [4] and to the continuous-state case in [25]. The effectiveness of such an approach, based on verification of so-called PCL-indexability conditions-as they are grounded on satisfaction by project performance metrics of partial conservation laws (PCLs)-has been demonstrated in diverse models.…”