Opportunistic Scheduling as Restless Bandits

Borkar, Vivek S.; Kasbekar, Gaurav S.; Pattathil, Sarath; Shetty, Priyesh Y.

doi:10.1109/tcns.2017.2774046

Cited by 36 publications

(28 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cross-layer control strategy to minimize communication latency under transmit power constraints have been studied in [27]- [34]. In [32], a Lazy scheduling policy that assigns scheduling decision based on the queue backlog is proposed.…”

Section: Related Workmentioning

confidence: 99%

Minimizing Age of Information With Power Constraints: Multi-User Opportunistic Scheduling in Multi-State Time-Varying Channels

Tang

Wang

Song

et al. 2020

IEEE J. Select. Areas Commun.

137

View full text Add to dashboard Cite

This work is motivated by the need of collecting fresh data from power-constrained sensors in the industrial Internet of Things (IIoT) network. A recently proposed metric, the Age of Information (AoI) is adopted to measure data freshness from the perspective of the central controller in the IIoT network. We wonder what is the minimum average AoI the network can achieve and how to design scheduling algorithms to approach it. To answer these questions when the channel states of the network are Markov time-varying and scheduling decisions are restricted to bandwidth constraint, we first decouple the multi-sensor scheduling problem into a single-sensor constrained Markov decision process (CMDP) by relaxing the hard bandwidth constraint. Next we exploit the threshold structure of the optimal policy for the decoupled single sensor CMDP and obtain the optimum solution through linear programming (LP). Finally, an asymptotically optimal truncated policy that can satisfy the hard bandwidth constraint is built upon the optimal solution to each of the decoupled single-sensor. Our investigation shows that to obtain a small AoI performance: (1) The scheduler exploits good channels to schedule sensors supported by limited power; (2) Sensors equipped with enough transmission power are updated in a timely manner such that the bandwidth constraint can be satisfied.

show abstract

Section: Related Workmentioning

confidence: 99%

Minimizing Age of Information With Power Constraints: Multi-User Opportunistic Scheduling in Multi-State Time-Varying Channels

Tang

Wang

Song

et al. 2020

IEEE J. Select. Areas Commun.

137

View full text Add to dashboard Cite

show abstract

“…Thus, while the MARBP is generally intractable, as it is known to be PSPACE-hard (see [35]), Whittle introduced, in [29], a widely applied heuristic index policy. For a sample of recent applications, see, for example [36][37][38][39][40][41][42][43][44][45][46][47][48]. Yet, the Whittle index is only defined for a limited class of restless bandits, called indexable, and it is nontrivial to verify whether such an indexability property holds for a given model.…”

Section: Approach Via Restless Bandit Reformulation Whittle Index Amentioning

confidence: 99%

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Niño‐Mora

2020

Mathematics

View full text Add to dashboard Cite

We consider the multi-armed bandit problem with penalties for switching that include setup delays and costs, extending the former results of the author for the special case with no switching delays. A priority index for projects with setup delays that characterizes, in part, optimal policies was introduced by Asawa and Teneketzis in 1996, yet without giving a means of computing it. We present a fast two-stage index computing method, which computes the continuation index (which applies when the project has been set up) in a first stage and certain extra quantities with cubic (arithmetic-operation) complexity in the number of project states and then computes the switching index (which applies when the project is not set up), in a second stage, with quadratic complexity. The approach is based on new methodological advances on restless bandit indexation, which are introduced and deployed herein, being motivated by the limitations of previous results, exploiting the fact that the aforementioned index is the Whittle index of the project in its restless reformulation. A numerical study demonstrates substantial runtime speed-ups of the new two-stage index algorithm versus a general one-stage Whittle index algorithm. The study further gives evidence that, in a multi-project setting, the index policy is consistently nearly optimal.

show abstract

“…The α parameter is chosen such that, subsidy η t is updated at a slower timescale compared to the value iteration algorithm that computes V S (π, η t ) and V N S (π, η t ). This is a two-timescale stochastic approximation algorithm and is based on similar schemes studied in [40], [41]. In [40,Chapter 6], the convergence of a two-timescale stochastic approximation algorithm was discussed.…”

Section: Algorithm For Whittle-index Computationmentioning

confidence: 99%

Sequential Decision Making With Limited Observation Capability: Application to Wireless Networks

Kaza

Meshram

Mehta

et al. 2019

IEEE Trans. Cogn. Commun. Netw.

View full text Add to dashboard Cite

This work studies a generalized class of restless multi-armed bandits with hidden states and allow cumulative feedback, as opposed to the conventional instantaneous feedback. We call them lazy restless bandits (LRB) as the events of decisionmaking are sparser than events of state transition. Hence, feedback after each decision event is the cumulative effect of the following state transition events. The states of arms are hidden from the decision-maker and rewards for actions are state dependent. The decision-maker needs to choose one arm in each decision interval, such that long term cumulative reward is maximized.As the states are hidden, the decision-maker maintains and updates its belief about them. It is shown that LRBs admit an optimal policy which has threshold structure in belief space. The Whittle-index policy for solving LRB problem is analyzed; indexability of LRBs is shown. Further, closed-form index expressions are provided for two sets of special cases; for more general cases, an algorithm for index computation is provided. An extensive simulation study is presented; Whittle-index, modified Whittleindex and myopic policies are compared. Lagrangian relaxation of the problem provides an upper bound on the optimal value function; it is used to assess the degree of sub-optimality various policies.

show abstract

Opportunistic Scheduling as Restless Bandits

Cited by 36 publications

References 37 publications

Minimizing Age of Information With Power Constraints: Multi-User Opportunistic Scheduling in Multi-State Time-Varying Channels

Minimizing Age of Information With Power Constraints: Multi-User Opportunistic Scheduling in Multi-State Time-Varying Channels

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Sequential Decision Making With Limited Observation Capability: Application to Wireless Networks

Contact Info

Product

Resources

About