Adaptive UAV-Trajectory Optimization Under Quality of Service Constraints: A Model-Free Solution

Abstract

Unmanned aerial vehicles (UAVs) with the potential of providing reliable high-rate connectivity, are becoming a promising component of future wireless networks. A UAV collects data from a set of randomly distributed sensors, where both the locations of these sensors and their data volume to be transmitted are unknown to the UAV. In order to assist the UAV in finding the optimal motion trajectory in the face of the uncertainty without the above knowledge whilst aiming for maximizing the cumulative collected data, we formulate a reinforcement learning problem by modelling the motion-trajectory as a Markov decision process with the UAV acting as the learning agent. Then, we propose a pair of novel trajectory optimization algorithms based on stochastic modelling and reinforcement learning, which allows the UAV to optimize its flight trajectory without the need for system identification. More specifically, by dividing the considered region into small tiles, we conceive state-action-reward-state-action (Sarsa) and Q-learning based UAV-trajectory optimization algorithms (i.e., SUTOA and QUTOA) aiming to maximize the cumulative data collected during the finite flight-time. Our simulation results demonstrate that both of the proposed approaches are capable of finding an optimal trajectory under the flight-time constraint. The preference for QUTOA vs. SUTOA depends on the relative position of the start and the end points of the UAVs.

Intelligent Trajectory Design for Secure Full- Duplex MIMO-UAV Relaying Against Active Eavesdroppers: A Model-Free Reinforcement Learning Approach

“…To that end, in [41], the authors have proposed an RL algorithm for a multi-UAV cooperative system, aiming at maximizing the sum rate metric via trajectory design and resource management. The authors in [42] have considered exploiting RL algorithms to optimize UAV's trajectory for maximum data collection in a sensor network under some QoS constraints. However, these developments have aimed at only reliability aspects of UAV communications, and PLS security aspects have not yet been fully explored.…”

Multiagent Q-learning based UAV trajectory planning for effective situational awareness

“…A paucity of contributions on the trajectory design of multiple UAVs leveraging multi-agent Reinforcement Learning (RL) has been proposed within the scope of several applications, such as resource allocation including sum transmit rate maximization, and the selection strategies of user, power level and subchannel [8,11,12], efficient spectrum management [1], fast and accurate localization [2,13] and maximum data collection [14].…”

“…Cui et al [14] formulated a RL problem for solving optimal trajectory design with the goal of maximized collected data in the presence of uncertainty factors under flight-time constraints. Finally, a compelling application of UAVs performing SAR missions in a post-disaster scenario can be observed in [19], where a UAV is navigated towards a wireless signal source, based on the received signal strength (RSS) levels fed to RL algorithm, that is attached to the victim without relying on GPS coordinates.…”

Heterogeneous Traffic Offloading in Space-Air-Ground Integrated Networks

“…Therefore, we observe that a concrete and self-adaptive integration of space and air segments in the terrestrial network could help fulfill the complex and heterogeneous slices of 5G while reducing considerably the service costs [5], [7]. The main integration approach in the literature is traffic offloading in SAGIN [9]- [14]. For instance, in [9], a sensor offloading framework was developed to design the UAVs trajectories while maximizing the collected data from different sensors and considering the limited energy on-board of UAVs.…”

“…For instance, satellites have the privileges of wide coverage and wide bandwidth. However, UAVs have the privileges of scalable deployment, mobility, low latency and reliability [8], [9]. Hence, space and air segments can support the terrestrial network to increase the users' connectivity and hence improve the service continuity, especially in under-served and rural areas.…”