Deep Reinforcement Learning for Fresh Data Collection in UAV-assisted IoT Networks

Abstract

Unmanned aerial vehicles (UAVs) have the potential to greatly aid Internet of Things (IoT) networks in missioncritical data collection, thanks to their flexibility and costeffectiveness. However, challenges arise due to the UAV's limited onboard energy and the unpredictable status updates from sensor nodes (SNs), which impact the freshness of collected data. In this paper, we investigate the energy-efficient and timely data collection in IoT networks through the use of a solarpowered UAV. Each SN generates status updates at stochastic intervals, while the UAV collects and subsequently transmits these status updates to a central data center. Furthermore, the UAV harnesses solar energy from the environment to maintain its energy level above a predetermined threshold. To minimize both the average age of information (AoI) for SNs and the energy consumption of the UAV, we jointly optimize the UAV trajectory, SN scheduling, and offloading strategy. Then, we formulate this problem as a Markov decision process (MDP) and propose a meta-reinforcement learning algorithm to enhance the generalization capability. Specifically, the compound-action deep reinforcement learning (CADRL) algorithm is proposed to handle the discrete decisions related to SN scheduling and the UAV's offloading policy, as well as the continuous control of UAV flight. Moreover, we incorporate meta-learning into CADRL to improve the adaptability of the learned policy to new tasks. To validate the effectiveness of our proposed algorithms, we conduct extensive simulations and demonstrate their superiority over other baseline algorithms.

Multi-UAV Path Planning for Wireless Data Harvesting With Deep Reinforcement Learning

“…The authors in [5] only investigate tablebased Q-learning for UAV data collection. A particular variety of IoT data collection is the one tackled in [10], where the authors propose a DQN-based solution to minimize the age of information of data collected from sensors. In contrast to our approach, the mentioned approaches are set in much simpler environments and agents have to undergo computationally expensive retraining when scenario parameters change.…”

“…Using the described UAV model in II-A and communication model in II-B, the central goal of the multi-UAV path planning problem is the maximization of throughput over the whole mission time and over all deployed UAVs while adhering to mobility constraints (6a)-(6e) and the scheduling constraint (10). The maximization problem is given by…”

“…We are able to solve the larger scenario without increasing network size, thanks to the global-local map approach. The value ranges from which the randomized scenario parameters are chosen as follows: number of deployed UAVs ∈ {1, 2, 3}, number of IoT sensors ∈ [3,10], data volume to be collected The performance on both maps is evaluated using Monte Carlo simulations on their respective full range of scenario parameters with overall average performance metrics shown in Table IV. Both agents show a similarly high successful landing performance.…”

“…The map has an order of magnitude more cells than the scenarios in [1]. The ranges for randomized scenario parameters are chosen as follows: number of deployed UAVs ∈ {1, 2, 3}, number of IoT sensors ∈ [5,10], data volume to be collected Fig. 6a shows a single agent trying to collect as much data as possible during the allocated maximum flying time.…”

“…works [5]- [9], with [10]- [13] concentrating on minimizing the age of information of the collected data. Additional coverage of past related work is offered in the next section.…”