Handover Control in Wireless Systems via Asynchronous Multiuser Deep Reinforcement Learning

Abstract

In this paper, we propose a two-layer framework to learn the optimal handover (HO) controllers in possibly large-scale wireless systems supporting mobile Internet-of-Things (IoT) users or traditional cellular users, where the user mobility patterns could be heterogeneous. In particular, our proposed framework first partitions the user equipments (UEs) with different mobility patterns into clusters, where the mobility patterns are similar in the same cluster. Then, within each cluster, an asynchronous multi-user deep reinforcement learning scheme is developed to control the HO processes across the UEs in each cluster, in the goal of lowering the HO rate while ensuring certain system throughput. In this scheme, we use a deep neural network (DNN) as an HO controller learned by each UE via reinforcement learning in a collaborative fashion. Moreover, we use supervised learning in initializing the DNN controller before the execution of reinforcement learning to exploit what we already know with traditional HO schemes and to mitigate the negative effects of random exploration at the initial stage. Furthermore, we show that the adopted global-parameter-based asynchronous framework enables us to train faster with more UEs, which could nicely address the scalability issue to support large systems. Finally, simulation results demonstrate that the proposed framework can achieve better performance than the state-of-art on-line schemes, in terms of HO rates.

Deep Reinforcement Learning-Based Modulation and Coding Scheme Selection in Cognitive Heterogeneous Networks

“…[25] adopted DRL to learn the power adaption strategy of the primary user in a cognitive network, such that the secondary user is able to adaptively control its power and satisfy the required quality of services of both primary and secondary users. [26] studied the handover problem in a multi-user multi-BS wireless network and proposed a DRL-based handover algorithm to reduce the handover rate of each user under a minimum sum-throughput constraint. In addition, [27] proposed a distributed DRLbased multiple access algorithm to improve the uplink sumthroughput in a multi-user wireless network.…”

Load Balancing for Ultradense Networks: A Deep Reinforcement Learning-Based Approach

“…Therefore, in this paper, we exploit the policy gradient method for policy improvement, which overcomes the limitations of the greedy searching strategy by explicitly optimizing a parameterized policy. Specifically, the policy gradient theorem [24] gives the analytic expression for the gradient of the objective J(π θ ) with respect to the policy parameters θ, written as (21) where d π θ is the state distribution when following policy π θ . Later, Silver et al proposed the off-policy deterministic policy gradient (OPDPG) theorem [25] to optimize a deterministic target policy by following a single stochastic behavior policy, written as…”

Dealing with Limited Backhaul Capacity in Millimeter-Wave Systems: A Deep Reinforcement Learning Approach

“…The general idea is to parameterize the Q-functions and derive the optimal values of parameters through policy gradient. In [10], [11], [13], the problems are formulated as multi-agent control with interactions among agents. As a result, experience replay for a single agent cannot be applied in such scenarios.…”