Delay-Optimal Resource Scheduling of Energy Harvesting-Based Devices

Abstract

This paper investigates resource scheduling in a wireless communication system operating with Energy Harvesting (EH) based devices and perfect Channel State Information (CSI). The aim is to minimize the packet loss that occurs when the buffer is overflowed or when the queued packet is older than a certain pre-defined threshold. We so consider a strict delay constraint rather than an average delay constraint. The associated optimization problem is modeled as Markov Decision Process (MDP) where the actions are the number of packets sent on the known channel at each slot. The optimal deterministic offline policy is exhibited through dynamic programming techniques, i.e. Value Iteration (VI) algorithm. We show that the gain in the number of transmitted packets and the consumed energy is substantial compared to: i) a naive policy which forces the system to send the maximum number of packets using the available energy in the battery, ii) two variants of the previous policy that take into account the buffer state, and iii) a policy optimized with an average delay constraint. Finally, we evaluate our optimal policy under imperfect CSI scenario where only an estimate of the channel state is available.

On Meeting a Maximum Delay Constraint Using Reinforcement Learning

“…However, efficient scheduling of arrived tasks with different priorities and task deadlines is also crucial. The authors in [9] consider a strict delay constraint for energy harvesting devices to find the best number of packets for transmission by modeling the problem as a Markov decision process (MDP). Solving the MDP through value iteration requires iterating over all states.…”

“…The value V θ is also used in order to estimate nodes' values. In this regard, we consider a linear combination of the super states' estimated value, calculated as stated in (9), and the neural network's output V θ to update nodes' values. In this way, as shown in Section VI, using VP-NN improves computation and enhances performance in two ways: first, improving the policy and estimated values in the tree; second, reducing both the depth and breadth of the search tree.…”