DQ Scheduler: Deep Reinforcement Learning Based Controller Synchronization in Distributed SDN

Abstract

In distributed software-defined networks (SDN), multiple physical SDN controllers, each managing a network domain, are implemented to balance centralized control, scalability and reliability requirements. In such networking paradigm, controllers synchronize with each other to maintain a logically centralized network view. Despite various proposals of distributed SDN controller architectures, most existing works only assume that such logically centralized network view can be achieved with some synchronization designs, but the question of how exactly controllers should synchronize with each other to maximize the benefits of synchronization under the eventual consistency assumptions is largely overlooked. To this end, we formulate the controller synchronization problem as a Markov Decision Process (MDP) and apply reinforcement learning techniques combined with deep neural network to train a smart controller synchronization policy, which we call the Deep-Q (DQ) Scheduler. Evaluation results show that DQ Scheduler outperforms the antientropy algorithm implemented in the ONOS controller by up to 95.2% for inter-domain routing tasks.

Bayesian Online Learning for Energy-Aware Resource Orchestration in Virtualized RANs

“…Similarly, online convex optimization is used for cloud and IoT resource orchestration [20], [21], but requires convex functions; a condition not satisfied here. Another approach is reinforcement learning (RL), used in spectrum management [14], network diagnostics [23], interference coordination [24], and SDN control [25], among others. However, RL suffers from the curse of dimensionality, and lacks convergence guarantees.…”

Optimizing the Lifetime of Software Defined Wireless Sensor Network via Reinforcement Learning

“…MIND predicts the spatial-temporal traffic information or network conditions, where the policy generation module is designed by optimal routing policies that learn from data based on RL. In distributed SDN, the problem of controller synchronization as a Markov Decision Process (MDP) was investigated in [28] with a limited synchronization budget to determine the rules that support the benefits of controller synchronization over time. An RL-based algorithm uses the deep neural network (DNN) to represent its value function, called the Deep-Q (DQ) Scheduler, which provides nearly twofold performance improvement compared to the state-ofthe-art SDN controller synchronization solutions.…”

Intelligent and Resizable Control Plane for Software Defined Vehicular Network : A Deep Reinforcement Learning Approach

“…Experiments in dynamic environments such as vehicular networks are missing to prove its efficiency in such challenging conditions. Authors in [12] were interested in distributed SDN and dealt with the synchronization problem between controllers based on a Deep Reinforcement Learning (DRL) approach. They propose a routing-focused DQ (Deep-Q) Scheduler to get the best policy that optimizes controller synchronization scheme over a time period.…”