Endpoint-Flexible Coflow Scheduling Across Geo-Distributed Datacenters

Li, Wenxin; Yuan, Xu; Li, Keqiu; Qi, Heng; Zhou, Xiaobo; Xu, Renhai

doi:10.1109/tpds.2020.2992615

Cited by 17 publications

(15 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Placing tasks close to their intermediate data for optimizing the completion times of jobs is called endpoint placement [7]. Iridium [13] and Flutter [14] reduce the job completion time by placing reduce tasks on or close to datacenters that have a large amount of intermediate data and relatively high link bandwidth.…”

Section: Related Workmentioning

confidence: 99%

“…The most related works to this papers are OMCoflow [9], [35], which combines scheduling and routing to minimize average CCT, and SmartCoflow [7], which combines scheduling and endpoint placement to minimize average CCT. For example, Li [35] proposes OneCoflow and OM-Coflow, two online algorithms, to route and schedule coflows simultaneously.…”

Section: Related Workmentioning

confidence: 99%

“…The coflow completion time (CCT) of coflow k associated with each coflow k is assumed to be known as soon as this coflow arrives: it includes the number of tasks to be launched on, where we use U i to denote the capacity of available computing slots in datacenter i ∈ N . Note that when a coflow k comes up and one of its task r is located on some datacenter i, it's assumed that people know how many flows will be created, the size of each flow, and the set of available paths between coflow k and datacenter i [7], [9]. To indicate the task placement and coflow routing, define I k,i r,f,p as whether the rth task associated with coflow k ∈ K is placed on datacenter i ∈ N and has a flow f ∈ F k,i r on path p ∈ P f .…”

Section: A System Modelmentioning

confidence: 99%

“…However, in the geo-distributed networks, the flows within a coflow have to traverse the heterogeneous inter-datacenter links, where several critical factors have impacts on the transmission. First, endpoints of flows in a coflow are closely correlated to the tasks of the corresponding job, while each task can be placed in any datacenter that has available computational resources [7]. Second, the bandwidth on the inter-datacenter links is limited and can vary significantly across different links [8].…”

Section: Introductionmentioning

confidence: 99%

“…Proof. Let t > 0 be a parameter and can be fixed later as in [7]. Then , where the Markov's inequality ensures the third transformation [9].…”

mentioning

confidence: 99%

See 4 more Smart Citations

Joint Online Coflow Optimization Across Geo-Distributed Datacenters

2020

IEEE Access

View full text Add to dashboard Cite

Growing data volumes have been generated, stored and processed across geographically distributed datacenters. Processing such geo-distributed datasets should consider task or job placement, heterogeneous available link bandwidth, and routing decisions. Thus optimizing the transmissions of inter-datacenter flows, especially coflows that capture application-level semantics, is important. However, prior solutions on coflow scheduling either merely ignore routing or fix the endpoint placement. This article focuses on the problem of jointly considering endpoint placement, coflow scheduling, and coflow routing to minimize the average coflow completion time (CCT) across geo-distributed datacenters. This paper first presents the system model and problem formulation, then derives a randomized approximation algorithm for a single coflow. After that it also proposes an online algorithm to handle the complex cases of multiple coflows. Through theoretical analysis, a proof that the proposed algorithms have a non-trivial competitive ratio is given. Results from extensive simulations demonstrate that the proposed algorithms can significantly reduce the CCT of coflows and at the same time, have similar algorithm run time, compared with the state-of-the-art solutions. INDEX TERMS Scheduling algorithms, routing, task placement, performance analysis.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: A System Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Proof. Let t > 0 be a parameter and can be fixed later as in [7]. Then , where the Markov's inequality ensures the third transformation [9].…”

mentioning

confidence: 99%

See 3 more Smart Citations

Joint Online Coflow Optimization Across Geo-Distributed Datacenters

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

FMap: A fuzzy map for scheduling elephant flows throughjumping traveling salesman problemvariant towardsoftware‐defined networking‐baseddata center networks

Abdollahi

Asadi

Montazerolghaem

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryNowadays data center networks (DCNs) should handle the ever‐growing load generated by diverse applications. It particularly occurs under concurrent flow requests and so‐called mice flows (MFs): elephant flows (EFs) ratio. Concentrating on a binary vision over flow size classification (EF or MF) results in subsequent unpredicted load imbalance due to neglecting EF's distinctions including a wide range of sizes. As a result, some EFs might utilize a path owing qualities beyond the given EF's demands, while another EF with higher requirements is attending to use an over‐utilized path. This article proposes FMap, a fuzzy map for scheduling EFs through our proposed variant of traveling salesperson problem (TSP) toward DCNs. FMap represents a novel EF scheduling scheme that integrates flow prioritization and routing decisions under the event pf parallel incoming flows besides the cooperation of the controller and OpenFlow switches in software‐defined networking (SDN) paradigm. FMap adopts fuzzy inference process to overcome the vagueness over EF's resource allocation. Mainly, FMap proposes a new variant of TSP (optimized by genetic algorithm) which enables EF's group forwarding with a minimum cost. FMap reduces the total hop count of EFs through considering a single optimal path for delivering groups of EFs that contain a same tag (priority). The outstanding results represent a major improvement as compared with equal cost multiple path, Hedera, Sonoum, and Size‐KP‐PSO. Particularly, the results illustrate an outperforming by 3.76×, 0.21×, 0.15×, 0.03×, and 0.03× in terms of total hop count, EFs FCT, packet loss, goodput, and received packets as compared with Size‐KP‐PSO, respectively.

show abstract