Power Allocation in Multi-User Cellular Networks with Deep Q Learning Approach

Abstract

The model-based power allocation algorithm has been investigated for decades, but it requires the mathematical models to be analytically tractable and it usually has high computational complexity.Recently, the data-driven model-free machine learning enabled approaches are being rapidly developed to obtain near-optimal performance with affordable computational complexity, and deep reinforcement learning (DRL) is regarded as of great potential for future intelligent networks. In this paper, the DRL approaches are considered for power control in multi-user wireless communication cellular networks.Considering the cross-cell cooperation, the off-line/on-line centralized training and the distributed execution, we present a mathematical analysis for the DRL-based top-level design. The concrete DRL design is further developed based on this foundation, and policy-based REINFORCE, value-based deep Q learning (DQL), actor-critic deep deterministic policy gradient (DDPG) algorithms are proposed.Simulation results show that the proposed data-driven approaches outperform the state-of-art modelbased methods on sum-rate performance, with good generalization power and faster processing speed.Furthermore, the proposed DDPG outperforms the REINFORCE and DQL in terms of both sum-rate performance and robustness, and can be incorporated into existing resource allocation schemes due to its generality.Deep reinforcement learning, deep deterministic policy gradient, policy-based, interfering multipleaccess channel, power control, resource allocation. I. INTRODUCTIONWireless data transmission has experienced tremendous growth in past years and will continue to grow in the future. When large numbers of terminals such as mobile phones and wearable devices are connected to the networks, the density of access point (AP) will have to be increased. Dense deployment of small cells such as pico-cells, femto-cells, has become the most effective solution to accommodate the critical demand for spectrum [1]. With denser APs and smaller cells, the whole communication network is flooded with wireless signals, and thus the intra-cell and inter-cell interference problems are severe [2]. Therefore, power allocation and interference management are crucial and challenging [3], [4].Massive model-oriented algorithms have been developed to cope with interference management [5]- [9], and the existing studies mainly focus on sub-optimal or heuristic algorithms, whose performance gaps to the optimal solution are typically difficult to quantify. Besides, the mathematical models are usually assumed to be analytically tractable, but these models are not always accurate because both hardware and channel imperfections can exist in practical communication environments. When considering specific hardware components and realistic transmission scenarios, such as low-resolution A/D, nonlinear amplifier and user distribution, the signal processing techniques with model-driven tools are challenging to be developed. Moreover, the computational complexity of these algorithms i...

Optimal Wireless Resource Allocation With Random Edge Graph Neural Networks

“…As per (20) we know that the state distributions satifym(Π T H Π, Π T x) = m(H, x). Substituting these two facts into (27) leads tô…”

“…Proposition 1 Consider wireless networks defined by probability distributions m(H, x) andm(Ĥ,x) along with resource allocations p andp. Assume there exists a permutation matrix Π for which (20) holds and that for the same permutation matrix…”

“…Perhaps the more compelling justification for statistical learning solutions to wireless resource allocation problems lies in their reliance on data over models. That is, by treating the resource allocation problem itself as a form of statistical regression, we obtain a means of training policies that solve the optimal allocation problem directly rather than via a training set [15]- [20]. This unsupervised approach exceeds the capabilities of supervised learning in that it can be applied to any arbitrary resource allocation problem and has the potential to exceed performance of existing heuristics.…”

An efficient Deep reinforcement learning with extended Kalman filter for device‐to‐device communication underlaying cellular network

“…In the work of Mao et al, 14 authors applied the Deep RL technique in machine intelligence, where the proposed system learns directly the resource management between various electronic devices by accumulating the experiences received within a cluster. In the work of Meng et al, 19 the authors addressed a Deep RL approach for allocating power to the devices in the multicellular network. Le et al 15 proposed a Deep RL-based offloading mechanism, where a user learns through an online trial and error basis to offload his computational job to other mobile devices with the help of D2D communication in an underlay cellular network.…”

“…To control the interference toward BS, they formulated a Deep Q-learning cost function using the Lagrangian method. In the work of Meng et al, 19 the authors addressed a Deep RL approach for allocating power to the devices in the multicellular network. The approach involves a mathematical analysis of Deep RL, considering the online/offline centralized training, intercell coordination and execution in a distributed fashion.…”