A Continuous Actor–Critic Deep Q-Learning-Enabled Deployment of UAV Base Stations: Toward 6G Small Cells in the Skies of Smart Cities

Abstract

Uncrewed aerial vehicle-mounted base stations (UAV-BSs), also know as drone base stations, are considered to have promising potential to tackle the limitations of ground base stations. They can provide cost-effective Internet connection to es that are out of infrastructure. They can also take over quickly as service providers when ground base stations fail in an unanticipated manner. UAV-BSs benefit from their mobile nature that enables them to change their 3D locations if the demand profile changes rapidly. In order to effectively leverage the mobility of UAV-BSs so as to maximize the performance of the network, 3D location of UAV-BSs requires continuous optimization. However, solving the optimization problem of UAV-BSs is NP-hard with no deterministic solution in polynomial time. In this paper, we propose a continuous actor-critic deep reinforcement learning solution in order to solve the location optimization problem of UAV-BSs in the presence of mobile endpoints. The simulation results show that the proposed model significantly improves the network performance compared to Qlearning, deep Q-learning and conventional algorithms. While the Q-learning and deep Q-learning-based baselines reach the sum data rate of 35 Mbps and 42 Mbps respectively, our proposed ACDQL-based strategy maximizes the sum data rate of endpoints to 45 Mbps. Furthermore, the proposed ACDQLbased methodology reduces the convergence time of the UAV-BS placement optimization by 85 percent compared to the Q-learning and deep Q-learning baselines.

Location and User Association Optimization in Multiple Radio Access UAV-Assisted Heterogeneous IoT Networks

“…The first one is to consider an LTE UAV instead of the Multi-RAT UAV. LTE UAVs have been considered in many works as in [29], [30], [54].…”

6G—Enabling the New Smart City: A Survey

“…The advent of consumer drones, flying ad hoc networks, low-latency 5G, and advancements beyond 5G have significantly accelerated progress in this field. The authors of [118] describe a continuous actor-critic deep Q-learning (ACDQL) strategy to solve the location optimization problem of UAV-BSs in the presence of mobile endpoints, extending the action space of the reinforcement learning (RL) algorithm from discrete to continuous. In detail, a scheme for the dynamic positioning of a UAV-BS in the case of user mobility is proposed, overcoming previous works presented in the literature, which considered fixed locations for the ground endpoints.…”

Toward Autonomous Multi-UAV Wireless Network: A Survey of Reinforcement Learning-Based Approaches

“…Q-learning is applied to select patches for leader UAVs, and follower UAVs are dispatched to cover the patches based on a star communication topology. Modifies the TD3 model for online path planning of multiple UAVs Energy efficiency maximization TD3 [173] Uses a DRL approach with flow-level models to determine optimal UAVs' trajectories Network throughput maximization, flow blocking probability minimization PPO [34] Presents a constrained DQN for optimizing the 3D trajectory design of multiple UAVs in a wireless network Downlink capacity maximization DQN [174] Using Q-learning based method to optimize the deployment and dynamic movement of UAVs Mean opinion score of ground users maximization Q-learning [175] Introduces a UAVs cell-free network utilizing DRL to optimize UAV trajectories in limited energy resources and insufficient environment knowledge Vehicle coverage maximization, energy consumption minimization DDPG [176] Presents a MADDPG-absed method for optimizing the trajectory of UAVs in a dynamic communication system Fairness of ground users, energy efficiency, throughput maximization MADDPG [177] Utilizes single-/multi-agent AC algorithms for initial UAV deployment and trajectory design Data downlink rate of users maximization A2C [178] Using heterogeneous graphs to represent the relationship between UAVs and users, and enhance the cooperation of multi-agent through communication Fair throughput maximization Multi-agent DQN [179] Leverage continuous AC deep Q-learning for optimal real-time UAV deployment Sum data rate maximization AC…”

“…In [179], the authors explore the deployment of UAV as aerial BSs to offer downlink Internet connectivity following the incapacitation of ground BS due to natural disasters. Given the mobility of ground endpoints, the authors emphasize the need for real-time deployment of UAVs to enhance network performance.…”