Efficient Scheduling of Weighted Coflows in Data Centers

Abstract

Traditional network resource management mechanisms are mainly flow or packet based. Recently, coflow has been proposed as a new abstraction to capture the communication patterns in a rich set of data parallel applications in data centers. Coflows effectively model the application-level semantics of network resource usage, so high-level optimization goals, such as reducing the transfer latency of applications, can be better achieved by taking coflows as the basic elements in network resource allocation or scheduling. Although efficient coflow scheduling methods have been studied, in this paper, we advocate to schedule weighted coflows as a further step in this direction, where weights are used to express the importances or priorities of different coflows or their corresponding applications. We propose the Weighted Coflow Completion Time (WCCT) minimization problem and a ð2 À 2 nþ1 Þ-approximate optimal offline algorithm, where n is the concurrent number of coflows. We then design an information-agnostic online algorithm named IAOA to dynamically schedule coflows according to their weights and the instantaneous network condition. We also design and implement a coflow scheduling system named FlyTransfer, which can use the online algorithm as its scheduling method. We test the performance of FlyTransfer by trace-driven simulations as well as real deployment in openstack. Our evaluation results show that, compared to the latest information-agnostic coflow scheduling algorithms, FlyTransfer can reduce more than 40 percent of the WCCT, and more than 30 percent of the completion time for coflows with above-the-average level of importance. It even outperforms the most efficient clairvoyant coflow scheduling method by reducing around 30 percent WCCT, and 25-30 percent of the completion time for coflows with above-the-average importance, respectively.

A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing

“…Unless otherwise specified, we choose the number of ports P = 20, the average job arrival rate λ = 20 and the average number of coflows n = 8. In the experiment, we compare our algorithm with DeepWeave, both based on DRL and Varys [6] and IAOA [33] that were also used as popular non-ML baselines for comparison with DeepWeave [1]. In addition, we add an ablation experiment to compare the performance of the retrained model under the same settings, removing the self-attention layer.…”

“…2. IAOA (Information-Agnostic Online Algorithm) [33] formulates the weighted coflow completion time minimiza-tion problem and proposes a heuristic solution with an approximation factor of 2 to the optimal solution. However, IAOA did not consider the dependency between coflows in job DAGs 3.…”

Efficient Routing Protection Algorithm Based on Optimized Network Topology

“…The network topology can be represented by an undirected connected graph [24][25][26] G = (V , E, W ). Where V , E, W respectively represents the set of nodes, the set of links, and the set of costs of links in the network, E is the complement of E. For link (u, v) ∈ E in the network, w (u, v) represents the cost of the link.…”

Application-Oriented Network Scheduling With Metaflow

“…From an another angle, the flaws of coflow have also been discussed recently. In [28], the problem of scheduling weighted coflows is addressed, where weights are used to express the importances of different coflows. Tian et al argue that there are dependencies among coflows in the context of multistage jobs and propose an approximation algorithm [29].…”