Q-Learning Based Routing Protocol for Congestion Avoidance

Abstract

The end-to-end delay in a wired network is strongly dependent on congestion on intermediate nodes. Among lots of feasible approaches to avoid congestion efficiently, congestion-aware routing protocols tend to search for an uncongested path toward the destination through rule-based approaches in reactive/incident-driven and distributed methods. However, these previous approaches have a problem accommodating the changing network environments in autonomous and self-adaptive operations dynamically. To overcome this drawback, we present a new congestion-aware routing protocol based on a Q-learning algorithm in software-defined networks where logically centralized network operation enables intelligent control and management of network resources. In a proposed routing protocol, either one of uncongested neighboring nodes are randomly selected as next hop to distribute traffic load to multiple paths or Q-learning algorithm is applied to decide the next hop by modeling the state, Q-value, and reward function to set the desired path toward the destination. A new reward function that consists of a buffer occupancy, link reliability and hop count is considered. Moreover, look ahead algorithm is employed to update the Q-value with values within two hops simultaneously. This approach leads to a decision of the optimal next hop by taking congestion status in two hops into account, accordingly. Finally, the simulation results presented approximately 20% higher packet delivery ratio and 15% shorter end-to-end delay, compared to those with the existing scheme by avoiding congestion adaptively.

A Reinforcement Learning-Based Routing for Real-Time Multimedia Traffic Transmission over Software-Defined Networking

“…When the RL agent chooses the route with the minimum delay and packet-loss ratio during implementation, it should receive the highest reward value. Authors in [26] employed a Q-learning as a congestion-aware routing protocol over SDN called (QCAR). They utilised a set of parameters to denote the node and link status, node queue length, the hop count to the destination and re-transmitted packet rate.…”

“…This paper aims to address this issue by introducing a framework that realises improving the satisfaction degree of end-users towards multimedia services by utilising SDN characteristics with RL, which intelligently decides the best path for multimedia flows. The proposed scheme follows the approach presented in [26], in terms of directly learning the route hop-by-hop. It also considers four link-state metrics (i.e., available bandwidth, delay, jitter and packet loss rate) as parameters for the RL agent to deliver video streaming packets.…”

Beacon-Based Hybrid Routing Protocol for Large-Scale Unmanned Vehicle Ad Hoc Network

“…In network routing protocols, literature [7] comprehensively analyzed and simulated typical AODV, DSDV, and OLSR protocols, but has not proposed a routing protocol with higher efficiency according to the differences obtained by simulation. Literature [8] introduced a clustering algorithm into the DSDV protocol and modified the criterion of link quality in the protocol so as to improve the network throughput. However, this method still used the method of flooding in the process of maintenance and path discovery, and the problems of transmission delay were not solved.…”