Dynamic Markov Decision Policies for Delay Constrained Wireless Scheduling

Abstract

Abstract-We consider a one-hop wireless system with a small number of delay constrained users and a larger number of users without delay constraints. We develop a scheduling algorithm that reacts to time varying channels and maximizes throughput utility (to within a desired proximity), stabilizes all queues, and satisfies the delay constraints. The problem is solved by reducing the constrained optimization to a set of weighted stochastic shortest path problems, which act as natural generalizations of max-weight policies to Markov decision networks. We also present approximation results for the corresponding shortest path problems, and discuss the additional complexity and delay incurred as compared to systems without delay constraints. The solution technique is general and applies to other constrained stochastic decision problems.

Channel-Based Sampling Rate and Queuing State Control in Delay-Constraint Industrial WSNs

“…In [9], a delay-based Lyapunov function is investigated to achieve joint stability and utility optimization in a multiuser one-hop wireless system with timevarying reliability. This approach is further discussed in [10] in a one-hop wireless system constituted by users with or without delay constraints. In addition, the max-weight queuing policy is extended in [10] with Markov Decisions associated with delay constraints using Lyapunov drift and Lyapunov optimization theory.…”

“…This approach is further discussed in [10] in a one-hop wireless system constituted by users with or without delay constraints. In addition, the max-weight queuing policy is extended in [10] with Markov Decisions associated with delay constraints using Lyapunov drift and Lyapunov optimization theory. In the Markov decision process, the system state is characterized by the aggregation of the channel state information and queuing state information.…”

QoS-Aware Power-Efficient Scheduler for LTE Uplink

“…One way to solve this problem is to formulate it as a constrained MDP. Although general techniques exist to solve MDPs, they suffer from the curse-ofdimensionality problem [25], where the number of states grows vastly with the numbers of both users and bearers. Consequently, formulating and solving constrained MDPs is non-trivial as it deals with an extremely large number of transition probabilities.…”

“…Consequently, formulating and solving constrained MDPs is non-trivial as it deals with an extremely large number of transition probabilities. Therefore, approximated solutions are often provided [25]. Moreover, a comprehensive knowledge of the users' channel responses and arrival processes may be required to solve the MDP.…”

Distributed Scheduling in Multiple Access With Bursty Arrivals Under a Maximum Delay Constraint

“…Energy-delay tradeoffs for multiuser wireless links with online arrivals were considered in [28], [31]. In particular, [28] considers a wireless downlink with a separate queue for each receiver.…”