Packet Flow Capacity Autonomous Operation Based on Reinforcement Learning

Abstract

As the dynamicity of the traffic increases, the need for self-network operation becomes more evident. One of the solutions that might bring cost savings to network operators is the dynamic capacity management of large packet flows, especially in the context of packet over optical networks. Machine Learning, particularly Reinforcement Learning, seems to be an enabler for autonomicity as a result of its inherent capacity to learn from experience. However, precisely because of that, RL methods might not be able to provide the required performance (e.g., delay, packet loss, and capacity overprovisioning) when managing the capacity of packet flows, until they learn the optimal policy. In view of that, we propose a management lifecycle with three phases: (i) a self-tuned threshold-based approach operating just after the packet flow is set up and until enough data on the traffic characteristics are available; (ii) an RL operation based on models pre-trained with a generic traffic profile; and (iii) an RL operation with models trained for real traffic. Exhaustive simulation results confirm the poor performance of RL algorithms until the optimal policy is learnt and when traffic characteristics change over time, which prevents deploying such methods in operators’ networks. In contrast, the proposed lifecycle outperforms benchmarking approaches, achieving noticeable performance from the beginning of operation while showing robustness against traffic changes.

Autonomous Flow Routing for Near Real-Time Quality of Service Assurance

“…A key novel contribution of this paper is the design and evaluation of the DRL-based flow routing agent, responsible of managing the routing of a target flow. From the lessons learnt in [6], a number of modules and workflows are presented to allow efficient and robust DRL-based routing immediately after the flow is provisioned and operation starts. Note that the DRL engine is able to learn and improve actions as soon as it gets experience from operation.…”

“…The DRL agent consists of two inter-related blocks: i) the learning agent is in charge of learning the best actions to be taken based on the current state and the received reward; and ii) the environment that is in charge of computing the state based on the observed traffic and current combination of routes (proportions), as well as the obtained reward based on the measured e2e delay for the selected routes. Without loss of generality, we assume that the DRL agent operates under the life-cycle presented in [6], where models are pre-trained offline with generic data and refined during online learning, thus achieving both decision-making robustness and high performance from the beginning of flow operation. In Fig.…”

“…Without loss of generality, we assume that the initial model has been pretrained offline, reproducing a network with the same topology G and routes P as those requested. A generic input traffic, e.g., a sinusoidal daily pattern with some random variation [6], as well as constant background traffic according to configured maxima in G are also assumed. Note that to enable offline training, the specific traffic characteristics are not well known at that time.…”

“…Upon the reception of a flow provisioning request (req), the SDN controller executes Algorithm 1 and obtains the set of allowable routes P. Once the provisioning of the flow routing agent finishes, the flow operation phase starts using the pre-trained offline model that better fits with req parameters. Operation was simulated by injecting realistic traffic according to the flow models used in [6]. We assume that the following flow provisioning request: <s=R1, t=R3, xmax=70Gb/s, dmax=0.15ms, pmax=2>.…”

“…A possible solution is to use (Deep (D)) Reinforcement Learning (RL) [4], [5]. (D)RL has been proposed for solving problems that require real-time decision making, like the management of the capacity of packet links [6] or that of optical connections [7].…”