Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A Survey

Abstract

<div>Next generation wireless networks are expected to be extremely complex due to their massive heterogeneity in terms of the types of network architectures they incorporate, the types and numbers of smart IoT devices they serve, and the types of emerging applications they support. In such large-scale and heterogeneous networks (HetNets), radio resource allocation and management (RRAM) becomes one of the major challenges encountered during system design and deployment. In this context, emerging Deep Reinforcement Learning (DRL) techniques are expected to be one of the main enabling technologies to address the RRAM in future wireless HetNets. In this paper, we conduct a systematic in-depth, and comprehensive survey of the applications of DRL techniques in RRAM for next generation wireless networks. Towards this, we first overview the existing traditional RRAM methods and identify their limitations that motivate the use of DRL techniques in RRAM. Then, we provide a comprehensive review of the most widely used DRL algorithms to address RRAM problems, including the value- and policy-based algorithms. The advantages, limitations, and use-cases for each algorithm are provided. We then conduct a comprehensive and in-depth literature review and classify existing related works based on both the radio resources they are addressing and the type of wireless networks they are investigating. To this end, we carefully identify the types of DRL algorithms utilized in each related work, the elements of these algorithms, and the main findings of each related work. Finally, we highlight important open challenges and provide insights into several future research directions in the context of DRL-based RRAM. This survey is intentionally designed to guide and stimulate more research endeavors towards building efficient and fine-grained DRL-based RRAM schemes for future wireless networks.</div>

Distributed reinforcement learning for dynamic spectrum allocation in cognitive radio‐based internet of things

“…According to the last numerical comparisons of the RL and DRL algorithm complexities [81], the worst‐case computational complexity of the approaches, using RL used algorithms as Q‐learning, SARSA, Actor–Critic and Monte Carlo, is O(|S|*|A|), where S and A are the sizes of the space of states and actions, respectively, whereas the worst‐case complexity of DDQN, DQN, and duelling DQN‐based approaches is O(|S| 2 ). They show a dependence on the size of spaces of states.…”

“…To deal with scalability issues faced by RL‐based approaches, the approaches in Ref. [55–58, 63, 65–68] have employed DRL‐based approaches based on DQN with one or two NNs, DDQN, Duelling DQN, and LSTM‐based A2C to allocate resources to RAN and (fog) edge network slices [81].…”

“…In Ref. [80], the authors propose a single‐agent DQN algorithm based on Monte Carlo Tree Search to address the problem of dynamic spectrum sharing between 4G LTE and 5G NR (New Radio) systems [81]. So far, all the summarised approaches done on the recent network Frequency assignment (FA) employed its own discrete state space with a certain number and depending on the network configuration, whereas the reward function is a continuous function defined in terms of the QoS requirement, and channels' supplies, and others, such as if the reward is online traffic real reward or not and if it is online traffic prediction reward or not.…”

“…Related studies can be found in the following sources but not restricted to Ref. [7–87]. Each of the proposed techniques has its own advantages and disadvantages, and none of them can be considered on its own to fully accomplish the realisation of CR systems.…”

Evolution of NOMA Toward Next Generation Multiple Access (NGMA) for 6G

“…• Mathematical optimization or ML approaches? Next-generation wireless networks are more complicated due to their large-scale, versatile, and heterogeneous nature [98], [218]. Conventional mathematical optimization approach requires complete or quasi-complete knowledge of the wireless environment, which is non-trivial to obtain in heterogeneous scenarios.…”