Deep Q-Network Learning Based Downlink Resource Allocation for Hybrid RF/VLC Systems

Abstract

Developing high data rate systems to meet the requirements of fifth generation mobile systems has become crucial. Hybrid radio frequency/visible light communication (RF/VLC) has appeared as a promising mechanism for achieving this objective. In hybrid RF/VLC, data rate maximization is subject to constraints on bandwidth, power and the user association. The joint optimization problem of bandwidth, power and user association to maximize the data rate is non-concave and obtaining an optimal solution is difficult with conventional optimization algorithms. The existing solutions are based on a presumption of at least one optimization variable. In this paper, this issue has been overcome by solving the joint optimization problem in hybrid RF/VLC with a deep Q-network (DQN) learning based algorithm, which has been recognized as an efficient learning based mechanism for optimization. Our system model considers one RF and multiple VLC access points (APs). The idle APs are also incorporated in the system model. The application of DQN learning based algorithm is carried out by finding an optimal policy with the help of an action-value function. As the data sets for the considered system are large, a multi-layered network is used for approximating the action-value function estimator. Finally, a transfer learning based algorithm has been proposed for maximizing the total data rate of the system for the case of a newly entering user equipment (UE) that uses the information of the environment before the arrival of the new UE. Through simulations, it is found that our proposed algorithms can lead to an improvement of more than 10% and 54% in the achievable sum-rate and number of iterations for convergence respectively as compared to that obtained with existing conventional optimization algorithms.

Distributed DRL-Based Downlink Power Allocation for Hybrid RF/VLC Networks

“…The experience replay and transfer model allowed the system to converge earlier and provide better performance by utilizing each others' experiences. Another DQN-based resource allocation scheme is proposed in [15] for hybrid RF/VLC networks. The numerical results show that the sum rate is 10% higher for the proposed DQN-based method, and the number of iterations is 54% less compared to the conventional methods for convergence [15].…”

“…Another DQN-based resource allocation scheme is proposed in [15] for hybrid RF/VLC networks. The numerical results show that the sum rate is 10% higher for the proposed DQN-based method, and the number of iterations is 54% less compared to the conventional methods for convergence [15]. However, in both works [14], [15], the use of the DQN-based causes to lose precision due to discrete action space and the DQN-based algorithm is designed as a centralized controller, which brings additional complexity and communication overhead.…”

“…The proposed optimization problem ( 8) is nonconvex as a large number of users with QoS requirements and limited resources in the system. Conventional algorithms cannot solve this optimization problem due to complexity [7], [8], [15]. As we mentioned in Section II, there are some methods such as GA [7] or PSO [8] which can handle complexity; however, these methods are not robust against dynamic systems and require frequent re-iteration for each change in the system.…”

“…DRL-based solutions can be applied to problems with discrete and continuous state and action spaces along with multivariable optimization problems. The Deep Q-Networks (DQN) can interpret observations of the environment with better performance with the help of neural networks [14], [15]. DQN's continuous state space allows it to understand higher-dimension complex problems.…”

Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A Survey

“…Most recently, the authors in [181] present a multi-agent DQN-based algorithm to address the problem of joint optimization of bandwidth, power, and user association in hybrid RF/VLC systems. The APs are the agents whose action is discrete, representing the bandwidth, association function, and power level.…”