Scheduling coflows of multi-stage jobs under network resource constraints

Zeng, Yue; Ye, Baoliu; Tang, Bin; Guo, Songtao; Qu, Zhihao

doi:10.1016/j.comnet.2020.107686

Cited by 7 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Few works have also considered scheduling coflows in general network topologies with bandwidth constraints [32] [11] [33]. The work in [32] proposed an efficient online solution for the multiple online coflow routing and scheduling problem.…”

Section: Related Workmentioning

confidence: 99%

Multihop Offloading of Multiple DAG Tasks in Collaborative Edge Computing

Sahni

Cao

Yang

et al. 2021

IEEE Internet Things J.

View full text Add to dashboard Cite

Collaborative edge computing has become a popular paradigm where edge devices collaborate by sharing resources. Data dissemination is a fundamental problem in CEC to decide what data is transmitted from which device and how. Existing works on data dissemination have not focused on coflow scheduling in CEC, which involves deciding the order of flows within and across coflows at network links. Coflow implies a set of parallel flows with a shared objective. The existing works on coflow scheduling in data centers usually assume a non-blocking switch and do not consider congestion at different links in the multihop path in CEC, leading to increased coflow completion time (CCT). Furthermore, existing works do not consider multiple flow sources that cannot be ignored, as data can have duplicate copies at different edge devices. This work formulates the multisource coflow scheduling problem in CEC, which includes jointly deciding the source and flow ordering for multiple coflows to minimize the sum of CCT. This problem is shown to be NPhard and challenging as each flow can have multiple dependent conflicts at multiple links. We propose a source and coflowaware search and adjust (SCASA) heuristic that first provides an initial solution considering the coflow characteristics. SCASA further improves the initial solution using the source search and adjust heuristic by leveraging the knowledge of both coflows and network congestion at links. Evaluation done using simulation experiments shows that SCASA leads to up to 83% reduction in the sum of CCT compared to benchmarks without a joint solution.

show abstract

Section: Related Workmentioning

confidence: 99%

Multihop Offloading of Multiple DAG Tasks in Collaborative Edge Computing

Sahni

Cao

Yang

et al. 2021

IEEE Internet Things J.

View full text Add to dashboard Cite

show abstract

“…In a multi-stage job scenario, due to the dependency between coflows, minimizing CCT may not minimize JCT. To the best of our knowledge, only a few works [12], [13], [1], [25] considered the coflow scheduling problem in the case of multi-stage jobs. Aalo [12] is the first effective heuristic algorithm with the objective of minimizing JCT, which discusses coflow scheduling in multi-stage jobs, and proposes to prioritize coflows according to dependency orders to schedule coflow.…”

Section: Related Workmentioning

confidence: 99%

“…However, this approach cannot scale to job DAGs of arbitrary sizes and shapes, and ignores the online situation where multiple multi-stage jobs coexist in the network. Similarly, [25] is the first to study how to schedule multi-stage jobs under taking network resource constraints into account so as to minimize the total weighted JCT.…”

Section: Related Workmentioning

confidence: 99%

A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing

Wang¹,

Shen²

2021

Preprint

View full text Add to dashboard Cite

Coflow is a recently proposed networking abstraction to help improve the communication performance of data-parallel computing jobs. In multi-stage jobs, each job consists of multiple coflows and is represented by a Directed Acyclic Graph (DAG). Efficiently scheduling coflows is critical to improve the data-parallel computing performance in data centers. Compared with hand-tuned scheduling heuristics, existing work DeepWeave [1] utilizes Reinforcement Learning (RL) framework to generate highly-efficient coflow scheduling policies automatically. It employs a graph neural network (GNN) to encode the job information in a set of embedding vectors, and feeds a flat embedding vector containing the whole job information to the policy network. However, this method has poor scalability as it is unable to cope with jobs represented by DAGs of arbitrary sizes and shapes, which requires a large policy network for processing a high-dimensional embedding vector that is difficult to train. In this paper, we first utilize a directed acyclic graph neural network (DAGNN) to process the input and propose a novel Pipelined-DAGNN, which can effectively speed up the feature extraction process of the DAGNN. Next, we feed the embedding sequence composed of schedulable coflows instead of a flat embedding of all coflows to the policy network, and output a priority sequence, which makes the size of the policy network depend on only the dimension of features instead of the product of dimension and number of nodes in the job's DAG. Furthermore, to improve the accuracy of the priority scheduling policy, we incorporate the Self-Attention Mechanism into a deep RL model to capture the interaction between different parts of the embedding sequence to make the output priority scores relevant. Based on this model, we then develop a coflow scheduling algorithm for online multi-stage jobs. Our simulation results are based on the real trace of Facebook. Compared with a state-of-the-art approach, our model can shorten the average weighted job completion time by up to 40.42% and complete jobs at least 1.68 times faster. It also has better scalability and robustness.

show abstract

“…Y. Chen et al proposed the multi-hop coflow routing and scheduling (MCRS) strategy in the popular Spine-Leaf topology and allocated longer detour paths to coflows to alleviate link congestion [ 36 , 37 ], which is applicable to over-subscribed Spine-Leaf networks. The authors of [ 32 , 33 , 34 , 35 , 36 , 37 ] only focused on joint flow scheduling and routing cases at the coflow level, whereas Y. Zeng et al were the first to study the job-level case [ 38 ]. They formulated the multi-stage job joint scheduling and routing problem as a non-linear weighted JCT minimization problem and designed a polynomial-time Multi-stage Job Scheduling (MJS) algorithm.…”

Section: Introductionmentioning

confidence: 99%

Job-Deadline-Guarantee-Based Joint Flow Scheduling and Routing Scheme in Data Center Networks

Suo,

Ma,

Jiao

et al. 2023

Sensors

View full text Add to dashboard Cite

Many emerging Internet of Things (IoT) applications deployed on cloud platforms have strict latency requirements or deadline constraints, and thus meeting the deadlines is crucial to ensure the quality of service for users and the revenue for service providers in these delay-stringent IoT applications. Efficient flow scheduling in data center networks (DCNs) plays a major role in reducing the execution time of jobs and has garnered significant attention in recent years. However, only few studies have attempted to combine job-level flow scheduling and routing to guarantee meeting the deadlines of multi-stage jobs. In this paper, an efficient heuristic joint flow scheduling and routing (JFSR) scheme is proposed. First, targeting maximizing the number of jobs for which the deadlines have been met, we formulate the joint flow scheduling and routing optimization problem for multiple multi-stage jobs. Second, due to its mathematical intractability, this problem is decomposed into two sub-problems: inter-coflow scheduling and intra-coflow scheduling. In the first sub-problem, coflows from different jobs are scheduled according to their relative remaining times; in the second sub-problem, an iterative coflow scheduling and routing (ICSR) algorithm is designed to alternately optimize the routing path and bandwidth allocation for each scheduled coflow. Finally, simulation results demonstrate that the proposed JFSR scheme can significantly increase the number of jobs for which the deadlines have been met in DCNs.

show abstract

Scheduling coflows of multi-stage jobs under network resource constraints

Cited by 7 publications

References 31 publications

Multihop Offloading of Multiple DAG Tasks in Collaborative Edge Computing

Multihop Offloading of Multiple DAG Tasks in Collaborative Edge Computing

A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing

Job-Deadline-Guarantee-Based Joint Flow Scheduling and Routing Scheme in Data Center Networks

Contact Info

Product

Resources

About