Optimal Frequency Reuse and Power Control in Multi-UAV Wireless Networks: Hierarchical Multi-Agent Reinforcement Learning Perspective

Abstract

To overcome the problems caused by the limited battery lifetime in multiple-unmanned aerial vehicle (UAV) wireless networks, we propose a hierarchical multi-agent reinforcement learning (RL) framework to maximize the energy efficiency (EE) of UAVs by finding the optimal frequency reuse factor and transmit power. The proposed algorithm consists of distributed inner-loop RL for transmit power control of the UAV terminal (UT) and centralized outer-loop RL for finding the optimal frequency reuse factor. Specifically, the proposed algorithm adjusts these two factors jointly to effectively mitigate intercell interference and reduce undesired transmit power consumption in multi-UAV wireless networks. We show that, for this reason, the proposed algorithm outperforms conventional algorithms, such as a random action algorithm with a fixed frequency reuse factor and a hierarchical multi-agent Q-learning algorithm with binary transmit power controls. Furthermore, even in the environment where UTs are continuously moving based on the mixed mobility model, we show that the proposed algorithm can find the best reward when compared to conventional algorithms.INDEX TERMS Unmanned aerial vehicle, optimal frequency reuse, transmit power control, energy efficiency, hierarchical multi-agent Q-learning, multi-UAV wireless network.

3-D UAV Small Cell Base Station Positioning and Resource Allocation in Cellular Network: A Stochastic Optimization Approach

“…Potential data rate − Required data rate Required data rate , (28) where Potential data rate = ∑ i∈I ∑ j∈I|x i j =1 0.332 × snr j × β min…”

“…As each cell uses radio frequencies only within its boundaries, in cells with no overlapped coverage areas, radio frequencies can be reused. To date, several studies introduced frequency reuse techniques to mitigate intracell and intercell interferences when sharing frequency resources in wireless networks [26], [27], [28]. According to [26], strict fractional frequency reuse (FFR) partitions a cell area into spatial regions with different frequency reuse factors, while soft frequency reuse (SFR) divides a cell area into two regions: an inner region with all of the frequency resources and an outer region with only a fraction of the resources.…”

“…The authors of [27] maximize the inter-BS distance among cells with the same resource plan by varying the SFR levels in each cell. In [28] the optimal frequency reuse factor is found by a hierarchical multi-agent reinforcement learning framework to maximize the energy efficiency of UAVs.…”