A Predictive Q-Learning Algorithm for Deflection Routing in Buffer-less Networks

Abstract

Abstract-In this paper, we introduce a predictive Q-learning deflection routing (PQDR) algorithm for buffer-less networks. Qlearning, one of the reinforcement learning (RL) algorithms, has been considered for routing in computer networks. The RL-based algorithms have not been widely deployed in computer networks where their inherent random nature is undesired. However, their randomness is sought-after in certain cases such as deflection routing, which may be employed to ameliorate packet loss caused by contention in buffer-less networks.We compare the proposed algorithm with two existing reinforcement learning-based deflection routing algorithms. Simulation results show that the proposed algorithm decreases the burst loss probability in the case of heavy traffic load while it requires fewer deflections. The PQDR algorithm is implemented using the ns-3 network simulator.

Intelligent Deflection Routing in Buffer-Less Networks

“…Reinforcement learning, which has been implemented in the proposed NN-NDD and ENN-NDD algorithms, provides a systematic framework for processing the gathered information. Various other deflection routing protocols based on reinforcement learning [8]- [10], [30] employ the Q-learning algorithm or its variants.…”

“…Q-learningbased deflection routing algorithms do not provide a procedure to reselect the paths that have low Q-values as a consequence of transient network conditions. The PQDR algorithm [10] enables a node to recover and reselect such paths and improve its decision-making ability. The PQDR algorithm combines the Predictive Q-routing algorithm [3] and RLDRS to optimally deflect contending flows.…”

“…2) Learning Deficiency: Q-learning is the only learning algorithm that has been employed for the deflection routing [8]- [10], [30]. The learning algorithm used in PQDR [10] is a variant of Q-learning and, hence, it inherits some of its deficiencies.…”

“…The learning algorithm used in PQDR [10] is a variant of Q-learning and, hence, it inherits some of its deficiencies. Even though Q-learning guarantees eventual convergence to an optimal policy when finding the best action set, it does not efficiently use the gathered data.…”

“…4) The proposed NDD signaling algorithm with NN, ENN, and Q-learning-based decision-making modules was incorporated into three reinforcement learning-based deflection routing protocols named NN-NDD, ENN-NDD, and Q-NDD. We implement these protocols within the iDef framework and compare their performance with the existing Reinforcement Learning Deflection Routing Scheme (RLDRS) [9] and Predictive Q-learning Deflection Routing (PQDR) [10]. The remainder of this manuscript is organized as follows.…”