Optimizing Deadline-Driven Bulk Data Transfers in Overlay Networks

Abstract

Abstract-Deadline-driven bulk data transfers frequently occur in overlay networks running data-intensive, distributed workflow applications, such as grid and cloud environments. What distinguishes such transfers from other Internet traffic is that overlay nodes should cooperate towards the common goal of delivering all inter-dependent data timely, rather than follow individual, selfish goals. For such scenarios, we propose scheduling transfers in overlays in a globally optimal manner with respect to minimizing overall network congestion. Our optimization jointly addresses routing of transfers within the overlay and the time-domain scheduling of transfer bandwidths. We formally define and address the associated problem, the Bulk Data Routing and Transfer(BDRT) and present a linear programming-based solution to it, optimal in both routing and time domains. We additionally explore alternative approaches based on heuristic routing strategies, both oblivious and time-domain optimized. We evaluate these solutions via both PlanetLab trace-driven simulations and Internet transfer experiments, on the Intrigger wide-area grid and PlanetLab. Evaluation shows that our approach finds optimal solutions, based on estimations of job arrival times, deadlines and transfer volumes.

Dimensioning of the Store-and-Transfer WDM Network With Limited Node Storage Under the Sliding Scheduled Traffic Model

“…The already paid for off-peak capacity can be fully utilized with the proposed algorithms. [13] Proposes an algorithm to cooperate transfers globally to meet deadlines and reduce congestion of the network. Both the transfer rate and the path can vary at each time interval.…”

“…Storage has been used in IP networks to support bulk data transfers [11][12][13]. A time-expanded graph is created to ensure deadline guarantee of the transfers.…”

A tight bound on the speed-up through storage for quickest multi-commodity flows

“…A feasible solution to the quickest multi-commodity flow problem is a flow over time f with time horizon T that satisfies constraints (1), (2), and (3). The objective is to minimize the time horizon T .…”

“…Notice that the flow conservation constraints (2) allow flow to arrive at an intermediate node v, wait there for some time, and continue onwards (i. e., the term on the left hand side of (2) might be positive at times). If storage of flow is prohibited at intermediate nodes, we can simply enforce equality in (2) for v ∈ V \ {s i , t i } at all times θ. Flows over time obeying these strict flow conservation constraints are called flows over time without (intermediate) storage.…”

“…Multi-commodity flows over time are a useful tool for modelling problems that involve non-instantaneous transportation through some kind of network. Some recent examples for such problems include aircraft routing [1], cloud computing [2,3], road traffic networks [4], disaster management [5,6], evacuations [7], logistics [8,9], packet routing [10] and shared-ride systems [11].…”