Markovian Restless Bandits and Index Policies: A Review

Niño‐Mora, José

doi:10.3390/math11071639

Cited by 6 publications

(1 citation statement)

References 168 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since its introduction, research on the RMABP has generated a vast amount of literature, which was partially reviewed in [12]. Much work has addressed applications of Whittle's index policy in a wide variety of models, where numerical studies often report a near-optimal performance.…”

Section: Aoi Minimization Via Transmission Scheduling and Restless Ba...mentioning

confidence: 99%

Age of Information Cost Minimization with No Buffers, Random Arrivals and Unreliable Channels: A PCL-Indexability Analysis

Niño-Mora

2023

Mathematics

Self Cite

View full text Add to dashboard Cite

Over the last decade, the Age of Information has emerged as a key concept and metric for applications where the freshness of sensor-provided data is critical. Limited transmission capacity has motivated research on the design of tractable policies for scheduling information updates to minimize Age of Information cost based on Markov decision models, in particular on the restless multi-armed bandit problem (RMABP). This allows the use of Whittle’s popular index policy, which is often nearly optimal, provided indexability (index existence) is proven, which has been recently accomplished in some models. We aim to extend the application scope of Whittle’s index policy in a broader AoI scheduling model. We address a model with no buffers incorporating random packet arrivals, unreliable channels, and nondecreasing AoI costs. We use sufficient indexability conditions based on partial conservation laws previously introduced by the author to establish the model’s indexability and evaluate its Whittle index in closed form under discounted and average cost criteria. We further use the index formulae to draw insights on how scheduling priority depends on model parameters.

show abstract

Section: Aoi Minimization Via Transmission Scheduling and Restless Ba...mentioning

confidence: 99%

Age of Information Cost Minimization with No Buffers, Random Arrivals and Unreliable Channels: A PCL-Indexability Analysis

Niño-Mora

2023

Mathematics

Self Cite

View full text Add to dashboard Cite

show abstract

Whittle index approach to the multi-class queueing systems with convex holding costs and IHR service times

Aalto

2024

Math Meth Oper Res

View full text Add to dashboard Cite

We consider the dynamic scheduling problem in a single-server multi-class queueing system, where the accumulated holding costs of a customer is a convexly increasing function of the time that the customer spends in the system. This problem was already addressed by van Mieghem in his seminal paper (Ann Appl Probab 5:808–833, 1995). He launched the so-called generalized $$c\mu $$ c μ rule and proved its asymptotic optimality in the heavy traffic limit. The optimal scheduling problem with convex holding costs have also been studied in many other papers. However, the holding costs in these papers are typically count-convex, i.e., the aggregate holding cost rate related to a class of customers increases convexly as a function of the current number of customers in this class, which is clearly different from the time-convex holding costs introduced in van Mieghem (Ann Appl Probab 5:808–833, 1995) and studied also in the present paper. We utilize the Whittle index approach to develop a novel heuristic policy for the dynamic scheduling problem with time-convex holding costs. Instead of the original open version of the continuous-time scheduling problem, we apply the Whittle index approach to the corresponding discrete-time closed version of the problem. As the main theoretical contribution, we prove that, for IHR service times, the discounted discrete-time problem is indexable and also give an explicit expression for the Whittle index itself. A special challenge in our model is the two-dimensional state space with an infinite number of possible states. By utilizing the discrete-time results, we develop a novel index policy for the original continuous-time problem and demonstrate, by numerical simulations, that this Whittle index policy outperforms the generalized $$c\mu $$ c μ rule. Thus, while the generalized $$c\mu $$ c μ rule is asymptotically optimal in the heavy traffic limit, it can still be improved in normal traffic conditions.

show abstract