Towards 5G: A Reinforcement Learning-Based Scheduling Solution for Data Traffic Management

Abstract

Abstract-Dominated by delay-sensitive and massive data applications, radio resource management in 5G access networks is expected to satisfy very stringent delay and packet loss requirements. In this context, the packet scheduler plays a central role by allocating user data packets in the frequency domain at each predefined time interval. Standard scheduling rules are known limited in satisfying higher Quality of Service (QoS) demands when facing unpredictable network conditions and dynamic traffic circumstances. This paper proposes an innovative scheduling framework able to select different scheduling rules according to instantaneous scheduler states in order to minimize the packet delays and packet drop rates for strict QoS requirements applications. To deal with real-time scheduling, the Reinforcement Learning (RL) principles are used to map the scheduling rules to each state and to learn when to apply each. Additionally, neural networks are used as function approximation to cope with the RL complexity and very large representations of the scheduler state space. Simulation results demonstrate that the proposed framework outperforms the conventional scheduling strategies in terms of delay and packet drop rate requirements.

Enhancing User Fairness in OFDMA Radio Access Networks Through Machine Learning

“…We propose to use the actor-critic approach that makes use of two functions: a) value or critic function that keeps track of the value of the states and criticize the actions; b) action-value or actor function that aims to learn over time the best parameters to be applied in each state. As per original definition, the value function V : S → R is determined as follows [19]:…”

“…where, R t+1 = r(s, a); (γ t R t+1 ; t ≥ 0) is the accumulated reward value being averaged from state to state by the discount factor γ ∈ [0, 1]; s[0] is considered as random such that P(s[0] = s) > 0 holds for every s ∈ S. The action-value function Q : S × A → R 2 considers in addition that the first action a[0] of the whole process is randomly chosen, and then the function becomes [19]:…”

“…In order to update these functions in each state, we use the recursions from [19] and the value function becomes:…”

“…If the improvement step is preferred, then a random set of fairness parameters a ∈ A is used to enhance the NGMN fairness satisfaction. For an optimal learning, it is preferred to dynamically change the improvement and exploitation steps according to some probability distributions that are decided in advance [19].…”

Dynamic Scheduler Management Using Deep Learning

“…In this paper, we propose a new approach to manage distributed scheduling methods autonomously across multiple cells simultaneously, reducing the requirement for cell planning and increasing the flexibility of scheduling methods in the network. A distributed approach to scheduler management is proposed by Comşa et al [21], where a flexible scheduler uses interchangeable scheduling policies to best serve the QoS requirements of its users. To select from these policies, the authors suggest a reinforcement learning framework, where the QoS offered by each scheduling policy is estimated by an individual neural network.…”