Reinforcement Learning Based Scheduling Algorithm for Optimizing Age of Information in Ultra Reliable Low Latency Networks

Abstract

Age of Information (AoI) measures the freshness of the information at a remote location. AoI reflects the time that is elapsed since the generation of the packet by a transmitter. In this paper, we consider a remote monitoring problem (e.g., remote factory) in which a number of sensor nodes are transmitting time sensitive measurements to a remote monitoring site. We consider minimizing a metric that maintains a trade-off between minimizing the sum of the expected AoI of all sensors and minimizing an Ultra Reliable Low Latency Communication (URLLC) term. The URLLC term is considered to ensure that the probability the AoI of each sensor exceeds a predefined threshold is minimized. Moreover, we assume that sensors tolerate different threshold values and generate packets at different sizes. Motivated by the success of machine learning in solving large networking problems at low complexity, we develop a low complexity reinforcement learning based algorithm to solve the proposed formulation. We trained our algorithm using the stateof-the-art actor-critic algorithm over a set of public bandwidth traces. Simulation results show that the proposed algorithm outperforms the considered baselines in terms of minimizing the expected AoI and the threshold violation of each sensor.

Wireless Network Intelligence at the Edge

“…As a remedy, RL can be adopted to explore and learn the system state dynamics and improve the process of decision-making over time. In this regard, minimizing AoI of remote sensors with the aid of an RL-based scheduler is presented in [217].…”

Age of Information for Multicast Transmission With Fixed and Random Deadlines in IoT Systems

“…For energy harvesting networks, the optimal scheduling policy for age-energy tradeoff and the online scheduling policy for AoI minimization were proposed in [21] and [22], respectively. Moreover, the authors in [23] and [24] developed reinforcement learning (RL) based algorithms to minimize the average AoI for ultra-reliable lowlatency communication (URLLC) and multi-flow networks, respectively. To balance the tradeoff between the AoI and the sampling cost, the authors in [25] proposed two non-monetary trigger-and-punishment mechanisms to achieve social optimal for scenarios with complete and incomplete information, respectively.…”

Unsupervised Deep Learning for Ultra-Reliable and Low-Latency Communications

“…2) Optimal Bandwidth Allocation: With the optimal power allocation function, the optimal bandwidth allocated to each user can be found from the equality constraint in (9). Due to the expectation in (9) and the complex expression of the achievable rate in (8b), the property of (9) is hard to analyze. In concept, the achievable rate should increase with the bandwidth.…”

“…To improve the resource usage efficiency while ensuring the QoS of URLLC, various resource allocation problems have been investigated in the existing literature [3][4][5][6][7][8][9]. To ensure the packet error/loss probabilities and the queueing delay violation probability, the QoS constraint needs to be ensured for arbitrary large-scale channel gains.…”

“…With multi-user diversity, the trade-off between reliability and resource usage efficiency can be improved, but the performance gap of the heuristic policy to the optimal solution is unknown. In [8,9], reinforcement learning was employed to solve the multi-timescale optimization problems in URLLC, where channel allocation and scheduling policies were learned according to the states of packet loss rate and the age of information, respectively. However, the reliability was controlled by taking the packet loss as penalties in the rewards, and hence the reliability cannot be ensured.…”