Deep Q-Learning-Based Transmission Power Control of a High Altitude Platform Station with Spectrum Sharing

Abstract

A High Altitude Platform Station (HAPS) can facilitate high-speed data communication over wide areas using high-power line-of-sight communication; however, it can significantly interfere with existing systems. Given spectrum sharing with existing systems, the HAPS transmission power must be adjusted to satisfy the interference requirement for incumbent protection. However, excessive transmission power reduction can lead to severe degradation of the HAPS coverage. To solve this problem, we propose a multi-agent Deep Q-learning (DQL)-based transmission power control algorithm to minimize the outage probability of the HAPS downlink while satisfying the interference requirement of an interfered system. In addition, a double DQL (DDQL) is developed to prevent the potential risk of action-value overestimation from the DQL. With a proper state, reward, and training process, all agents cooperatively learn a power control policy for achieving a near-optimal solution. The proposed DQL power control algorithm performs equal or close to the optimal exhaustive search algorithm for varying positions of the interfered system. The proposed DQL and DDQL power control yields the same performance, which indicates that the actional value overestimation does not adversely affect the quality of the learned policy.

Handling Interference in Integrated HAPS-Terrestrial Networks through Radio Resource Management

“…In problem (6), constraints (6b) and (6c) limit the maximum transmit power of the MBS and HAPS. Constraint (6d) ensures that orthogonal subcarrier allocation has been preserved within each MBS and HAPS.…”

“…Due to the presence of binary variable F, and non-convex objective function and constraint (6d), problem ( 6) is a mixedinteger non-linear program (MINLP), which is challenging to solve for the global solution. Thus, in the following section, we develop a SCA-based rapid converging iterative algorithm to solve (6) for the sub-optimal solution. To this end, we apply the reformulation-linearization techniques (RLT) to transform the MINLP into an equivalent convex problem, which is then solved in an iterative manner until it converges to the sub-optimal solution.…”

“…For vHetNets, the challenge of interference gets more complicated and challenging, and urges for more novel techniques to deal with it. To this end, the authors in [6] proposed a multi-agent deep Q-learning (DQL)-based transmit power control algorithm to minimize the outage probability of the HAPS downlink channel under the constraint of interference tolerance of the terrestrial network. The authors in [7] designed transmit beamformers to mitigate interference from the HAPS to terrestrial cells.…”

A Multi-AUV Maritime Target Search Method for Moving and Invisible Objects Based on Multi-Agent Deep Reinforcement Learning

“…Jiang et al [ 21 ] propose a value-iteration-based RL algorithm, which can efficiently converge to stable strategies and significantly improve network performance. Jo et al [ 22 ] propose a multi-agent Deep Q-learning (DQL)-based transmission power control algorithm which minimizes the outage probability of the High-Altitude Platform Station downlink. However, existing multi-agent algorithms are capable of static target search tasks and have poor performance for moving and invisible objects.…”

Editorial: Special Issue “Unmanned Aerial Vehicle (UAV)-Enabled Wireless Communications and Networking”

“…Ref. [8] proposed a multi-agent DQL (Deep Q-learning)-based transmission power control algorithm to minimize the outage probability while satisfying the interference requirement of an interfered system. To deal with the potential risk of action-value overestimation from the DQL, they developed even a DDQL (Double DQL).…”