Online reinforcement learning for adaptive interference coordination

Abstract

Heterogeneous networks (HetNets), in which small cells overlay macro cells, are a cost-effective approach to increase the capacity of cellular networks. However, HetNets have raised new issues related to cell association and interference management. In particular, the optimal configuration of interference coordination (IC) parameters is a challenging task because it depends on multiple stochastic processes such as the locations of the users, the traffic demands, or the strength of the received signals. This work proposes a self-optimization algorithm capable of finding the optimal configuration in an operating network. We address the problem using a reinforcement learning (RL) approach, in which the actions are the IC configurations, whose performances are initially unknown. The main difficulty is that, due to the variable network conditions, the performance of each action may change over time. Our proposal is based on two main elements: the sequential exploration of subsets of actions (exploration regions), and an optimal stopping (OS) strategy for deciding when to end current exploration and start a new one. For our algorithm, referred to as local exploration with optimal stopping (LEOS), we provide theoretical bounds on its long-term regret per sample and its convergence time. We compare LEOS to state-of-the-art learning algorithms based on multiarmed bandits and policy gradient RL. Considering different changing rates in the network conditions, our numerical results show that LEOS outperforms the first alternative by 22%, and the second one by 48% in terms of average regret per sample.

Orchestrating Energy-Efficient vRANs: Bayesian Learning and Experimental Results

“…Similarly, online convex optimization is used for cloud and IoT resource orchestration [27], [28], but requires convex functions; a condition not satisfied here. Another approach is reinforcement learning (RL), used in spectrum management [16], network diagnostics [29], interference coordination [30], and SDN control [31], among others. In this line, [32], [33] optimize the energy efficiency of the network as a function of some parameters such as the resource block allocation, the transmission power, or the amount of network offloading.…”

Bayesian Online Learning for Energy-Aware Resource Orchestration in Virtualized RANs

“…Similarly, online convex optimization is used for cloud and IoT resource orchestration [20], [21], but requires convex functions; a condition not satisfied here. Another approach is reinforcement learning (RL), used in spectrum management [14], network diagnostics [23], interference coordination [24], and SDN control [25], among others. However, RL suffers from the curse of dimensionality, and lacks convergence guarantees.…”

Model-Based Reinforcement Learning With Kernels for Resource Allocation in RAN Slices

“…There are no precedents of this approach for dynamic resource allocation among network slices. Previous online learning proposals were focused on different network functions (e.g., interference coordination and energy saving [33]- [35]) and used specific ad hoc mechanisms based on multi-armed bandits [33], sequential likelihood ratio tests [34], or bayesian models [35].…”