A Decoupled Learning Strategy for Massive Access Optimization in Cellular IoT Networks

Abstract

Cellular-based networks are expected to offer connectivity for massive Internet of Things (mIoT) systems. However, their Random Access CHannel (RACH) procedure suffers from unreliability, due to the collision from the simultaneous massive access. Despite that this collision problem has been treated in existing RACH schemes, these schemes usually organize IoT devices' transmission and re-transmission along with fixed parameters, thus can hardly adapt to time-varying traffic patterns. Without adaptation, the RACH procedure easily suffers from high access delay, high energy consumption, or even access unavailability. With the goal of improving the RACH procedure, this paper targets to optimize the RACH procedure in real-time by maximizing a long-term hybrid multi-objective function, which consists of the number of access success devices, the average energy consumption, and the average access delay. To do so, we first optimize the long-term objective in the number of access success devices by using Deep Reinforcement Learning (DRL) algorithms for different RACH schemes, including Access Class Barring (ACB), Back-Off (BO), and Distributed Queuing (DQ). The converging capability and efficiency of different DRL algorithms including Policy Gradient (PG), Actor-Critic (AC), Deep Q-Network (DQN), and Deep Deterministic Policy Gradient (DDPG) are compared. Inspired by the results from this comparison, a decoupled learning strategy is developed to jointly and dynamically adapt the access control factors of those three access schemes. This decoupled strategy integrates predicted traffic into the learning process to improve training efficiency, where a Recurrent Neural Network (RNN) model is first employed to predict the real-time traffic values of the network environment, and then multiple DRL agents are employed to cooperatively configure parameters of each RACH scheme. Our results demonstrate that the decoupled strategy remarkably accelerate the training speedy. Index Terms-RACH, overload control, energy-delay tradeoff, traffic prediction, deep reinforcement learning. I. INTRODUCTION Cellular-based radio access technologies are required to support massive Internet of Things (mIoT) ecosystem, due to its Manuscript

Spectrum Demand Forecasting for IoT Services

“…Other interesting work in [27] has proposed a reinforcement learning-based mechanism on a Markov decision process and Monte Carlo Q-learning technique for modeling the network traffic in the implementation of network security and management functions in industrial IoT applications. Additional applications can be also found in the literature regarding novel radio access mechanisms combined with deep reinforcement learning techniques and the RNN traffic prediction model, which tackle the problem of collisions with massive IoT networks [28]. When using AI-based traffic prediction models, the main purpose is to obtain information about the network traffic in short periods of time for network management functions.…”

Machine Learning for Massive Industrial Internet of Things

“…Several MAC-layer problems can be formulated as POMDP: 1) to mitigate the serious collision in massive IoT access, DRL can be applied to optimize access control schemes. In [1], [12], DRL dynamically adapts the access control factors based on the traffic prediction via RNN. With multi-agent configuring the parameters of different schemes, the GNN has the potential to further enhance their cooperation; and 2) scheduling algorithms are important components to provide the guaranteed IIoT QoS requirements.…”

“…The reward in each slot is defined as the number of successful transmissions. DQN and DDPG are applied to optimize the ACB factor (θ ∈ (0, 1])) in the discrete (i.e., with the pace of 0.05) and continuous action space [12], respectively. Fig.…”

QoE Optimization for Live Video Streaming in UAV-to-UAV Communications via Deep Reinforcement Learning

“…However, in large search and rescue firefighting scenario, a nonordinary optical camera [24] should be considered to ensure the reception of a high quality video. To deal with a more complex environment and practical scenarios, such as search and rescue firefighting scenarios, the DRL algorithm is a promising tool for solving the problem of jointly optimizing the UAVs location while maximizing the data rate [25], [26]. DRL scheme has been applied to improve the performance of Vehicular Ad doc networks [27] and interference alignment problems in wireless networks [28].…”