Rapier: Integrating routing and scheduling for coflow-aware data center networks

Abstract

In the data flow models of today's data center applications such as MapReduce, Spark and Dryad, multiple flows can comprise a coflow group semantically. Only completing all flows in a coflow is meaningful to an application. To optimize application performance, routing and scheduling must be jointly considered at the level of a coflow rather than individual flows. However, prior solutions have significant limitation: they only consider scheduling, which is insufficient. To this end, we present RAPIER, a coflow-aware network optimization framework that seamlessly integrates routing and scheduling for better application performance. Using a smallscale testbed implementation and large-scale simulations, we demonstrate that RAPIER significantly reduces the average coflow completion time (CCT) by up to 79.30% compared to the state-of-the-art scheduling-only solution, and it is readily implementable with existing commodity switches.

A Coflow-Based Co-Optimization Framework for High-Performance Data Analytics

“…To optimize the performance of big data analytics, we need to optimize data flows transferred at the level of coflow rather than individual ones. This is because the completion time of a job (i.e., a distributed join) depends on the time it takes to complete the entire coflow, which is also called the coflow completion time (CCT), instead of the time to complete the individual flows composing it [10].…”

“…Regardless, without sacrificing generality, we assume that each individual data flow starts at the same time in this work. In the meantime, though detailed performance analysis of coflows in complex network environments (e.g., routing [10]) have been explored, we adopt an approach similar to that of Varys [8] and model the underlying network abstract as a non-blocking switch, which interconnects all the machines 1 . Moreover, we assume that all the network ports have the same normalized unit capacity, and bandwidth competition only appears in ingresses or egresses.…”

“…On the other hand, current research on coflow scheduling in data communication domain assumes that the detailed information (i.e., [src, des, v]) of each data flow is known before a coflow starts [8], [10], [13]. Namely, they have not realized that high-level scheduling of data movement could actually impact the final communication performance.…”

“…Current research on coflow scheduling aims at minimizing the average CCT cost and meeting coflow deadlines [8]. Advanced approaches and implementations, such as online conditions [9], with complex networks [10], have been proposed for practical environments. All of these methods are based on the assumption the source and destination of each data flow have been known.…”

“…Rather than individual flows, current work focuses on improving the performance of data flows for a job (e.g., a parallel join), such as minimizing the completion time of the slowest flow. To date, several solutions, such as Varys [8], Aalo [9] and RAPIER [10] have been shown to be very efficient on coflow processing, and thus they can be directly applied to current systems to speedup big data analytics. Regardless, all these techniques focus on network-level optimization, which is decoupled from application-level optimization/scheduling, and thus they could lead to suboptimal performance (we demonstrate this problem with more details in Section II).…”