Neural Combinatorial Deep Reinforcement Learning for Age-optimal Joint Trajectory and Scheduling Design in UAV-assisted Networks

Abstract

In this paper, an unmanned aerial vehicle (UAV)-assisted wireless network is considered in which a battery-constrained UAV is assumed to move towards energy-constrained ground nodes to receive status updates about their observed processes. The UAV's flight trajectory and scheduling of status updates are jointly optimized with the objective of minimizing the normalized weighted sum of Age of Information (NWAoI) values for different physical processes at the UAV. The problem is first formulated as a mixed-integer program. Then, for a given scheduling policy, a convex optimization-based solution is proposed to derive the UAV's optimal flight trajectory and time instants on updates. However, finding the optimal scheduling policy is challenging due to the combinatorial nature of the formulated problem.Therefore, to complement the proposed convex optimization-based solution, a finite-horizon Markov decision process (MDP) is used to find the optimal scheduling policy. Since the state space of the MDP is extremely large, a novel neural combinatorial-based deep reinforcement learning (NCRL) algorithm using deep Q-network (DQN) is proposed to obtain the optimal policy. However, for large-scale scenarios with numerous nodes, the DQN architecture cannot efficiently learn the optimal scheduling policy anymore. Motivated by this, a long short-term memory (LSTM)-based autoencoder is proposed to map the state space to a fixed-size vector representation in such large-scale scenarios while capturing the spatio-temporal interdependence between the update locations and time instants. A lower bound on the minimum NWAoI is analytically derived which provides system design guidelines on the appropriate choice of importance weights for different nodes. Furthermore, an upper bound on the UAV's minimum speed is obtained to achieve this lower bound value. The numerical results also demonstrate that the proposed NCRL approach can significantly improve the achievable NWAoI per process compared to the baseline policies, such as weight-based and discretized state DQN policies.

Age of Information in Practice

“…[49] formulated the AoI minimization problem on a point-to-point link through the application of HARQ, as a constrained Markov Decision Process and solved it by using Value Iteration and SARSA algorithms, on discrete state space (ages defined as integers). More recently, [50][51][52][53][54] considered the application of machine learning methods to age optimization in various network settings.…”

Reinforcement Learning in the Sky: A Survey on Enabling Intelligence in NTN-Based Communications

“…Accordingly, the UAV can choose to move to an adjacent cell in the next time slot or remain in its current position. This work was extended in [136] where a neural combinatorial-based deep RL algorithm was proposed using a DQN. To handle a very large number of nodes, a Long Short-Term Memory (LSTM) auto-encoder was used to reduce the dimensions of the state space to a fixed-length vector.…”

“…• Most of the works simulated a realistic wireless channel in their system model using the probabilistic path loss model [80,93,94,104,104,107,107,108,108,118,123,127,128,148,172,172,179,180] with only a few works achieving higher realism by considering CSI estimation [79,112,131,137,158]. • In terms of energy considerations, a fair number of works presented energy-efficient factors and constraints in their formulations such as battery capacity [103], energy harvesting [112,150,157], propulsion energy [113,139], energy quanta [136,139] and others [95,96,140]. Upon analyzing Table 5, we notice that more attention was paid to 3D environments with more realistic deployment scenarios where multiple non terrestrial platforms coordinate together to provide multi-user access control [116] in NTNs, space-air-ground integrated link optimization [151,153], maximizing end-to-end data rate [117] and others [154,155].…”