Reinforcement Learning for Datacenter Congestion Control

Abstract

We approach the task of network congestion control in datacenters using Reinforcement Learning (RL). Successful congestion control algorithms can dramatically improve latency and overall network throughput. Until today, no such learning-based algorithms have shown practical potential in this domain. Evidently, the most popular recent deployments rely on rule-based heuristics that are tested on a predetermined set of benchmarks. Consequently, these heuristics do not generalize well to newly-seen scenarios. Contrarily, we devise an RL-based algorithm with the aim of generalizing to different configurations of real-world datacenter networks. We overcome challenges such as partial-observability, nonstationarity, and multi-objectiveness. We further propose a policy gradient algorithm that leverages the analytical structure of the reward function to approximate its derivative and improve stability. We show that this scheme outperforms alternative popular RL approaches, and generalizes to scenarios that were not seen during training. Our experiments, conducted on a realistic simulator that emulates communication networks' behavior, exhibit improved performance concurrently on the multiple considered metrics compared to the popular algorithms deployed today in real datacenters. Our algorithm is being productized to replace heuristics in some of the largest datacenters in the world.

Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs

“…Another crucial component in the success of ADPG is its RTT-based reward that at its optimum depicts an optimal flow equilibrium. In line with our summary above, Tessler et al [44] state that for potential deployment, they would require dedicated hardware to accommodate the computational burden of deep-learning inference. In this work, we build upon [44] and devise a reward-modified variant which we call RL-CC.…”

“…Recently, Tessler et al [44] introduced an RL-based RDMA CC algorithm called Analytic Deterministic Policy Gradient (ADPG). In several network simulation benchmarks, ADPG outperformed SOTA rule-based CC algorithms: DCQCN, SWIFT, and HPCC.…”

“…These methods tend to perform exceptionally well in a specific set of tasks yet under-perform in others that they were not optimized for. For instance, DCQCN [53] and SWIFT [25] have been optimized for steady-state scenarios; but, as shown in [44], their reaction time is slow for sudden bursts of short flows. Recently, Tessler et al [44] introduced a data-driven approach to learning a CC policy.…”

“…For instance, DCQCN [53] and SWIFT [25] have been optimized for steady-state scenarios; but, as shown in [44], their reaction time is slow for sudden bursts of short flows. Recently, Tessler et al [44] introduced a data-driven approach to learning a CC policy. They presented a reward function (measuring the latency and throughput) and devised an algorithm that maximizes the cumulative reward throughout multiple steps.…”

“…Specifically, we map deep networks with 450 𝜇𝑠𝑒𝑐 latency to decision trees with 0.9 𝜇𝑠𝑒𝑐 latency. (2) We improve the optimization objective presented in [44], cast RTT dependence to switch queue utilization, and analyze the controllable reward parameters. (3) We deploy our method on NVIDIA production NICs, ConnectX-6Dx, in an operational cluster of 64 hosts, and run extensive evaluations reaching SOTA results.…”