Data placement in widely distributed environments

Kosar, Tevfik; Son, Sechang; Kola, George; Livny, Miron

doi:10.1016/s0927-5452(05)80008-5

Cited by 12 publications

(12 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Parallelism sends different chunks of the same file over parallel data streams (typically TCP connections), and can achieve high throughput by aggregating multiple streams and utilizing a larger share of the available network bandwidth [9], [11], [21], [22]. Concurrency refers to sending multiple files simultaneously through the network using different data channels, and is especially useful for increasing I/O concurrency in parallel disk systems [23], [24], [25]. Figure 1 shows the impact of protocol parameters concurrency and pipelining on transfer throughput.…”

Section: Motivationmentioning

confidence: 99%

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

Arslan

Kosar

2018

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Data-intensive scientific and commercial applications increasingly require frequent movement of large datasets from one site to the other(s). Despite growing network capacities, these data movements rarely achieve the promised data transfer rates of the underlying physical network due to poorly tuned data transfer protocols. Accurately and efficiently tuning the data transfer protocol parameters in a dynamically changing network environment is a major challenge and remains as an open research problem. In this paper, we present predictive end-to-end data transfer optimization algorithms based on historical data analysis and real-time background traffic probing, dubbed HARP. Most of the previous work in this area are solely based on real time network probing which results either in an excessive sampling overhead or fails to accurately predict the optimal transfer parameters. Combining historical data analysis with real time sampling enables our algorithms to tune the application level data transfer parameters accurately and efficiently to achieve close-to-optimal end-to-end data transfer throughput with very low overhead. Our experimental analysis over a variety of network settings shows that HARP outperforms existing solutions by up to 50% in terms of the achieved throughput.

show abstract

Section: Motivationmentioning

confidence: 99%

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

Arslan

Kosar

2018

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…This is even more crucial when a big data workflow is executed in a distributed environment that involves multiple and heterogeneous data centers. Kosar et al, (2005aKosar et al, ( , 2005b proposed an allocation framework for distributed computing systems which considered the data placement subsystem as an independent module along with the computation subsystem. In their proposed model, both data placement and task computation jobs can be queued, scheduled, monitored, managed and even check pointed.…”

Section: Related Workmentioning

confidence: 99%

Task And Data Allocation Strategies for Big Data Workflows

Ebrahimi¹,

Mohan²,

Kashlev³

et al. 2015

STBD

View full text Add to dashboard Cite

The makespan of a big data workflow is the time elapsed between the start of the first task and the completion of the last task in the workflow. This time includes the delivery of the final data product to the desired location within the network. Due to the large number of inputs and intermediate outputs of a big data workflow activity, the makespan of the workflow is significantly influenced by how its tasks and datasets are allocated in a distributed computing environment. Therefore, reducing makespans of big data workflows can be achieved by incorporating a data and task allocation strategy into an execution planning phase performed by a workflow management system. This creates a pressing need for an investigation of such strategies. To address this need, this paper provides a formal definition of the makespan minimization problem for big data workflows and proposes efficient workflow execution planning strategies. In particular, two algorithms, WEP-A and WEP-B, following different strategies are proposed. WEP-A follows a phased approach to the generation of an execution plan whereas WEP-B uses an evolutionary algorithm-based optimization strategy to find a valid plan with the shortest makespan. Both of these strategies are evaluated through extensive simulation experiments by varying workflow graphs and resources in the workflow environment. The results of the experiments demonstrate that WEP-B performs better than WEP-A on a set of benchmark examples. For more complex and large workflows, the improvements due to evolutionary optimization in WEP-B are likely to be even more pronounced.

show abstract

“…The Stork data scheduler [24,37,38] implements techniques specific to queuing, scheduling and optimization of data placement jobs, provides high reliability in data transfers and creates a level of abstraction between the user applications and the underlying data transfer and storage resources (including file transfer protocol (FTP), hypertext transfer protocol (HTTP), GridFTP, SRM, SRB and iRODS) via a modular, uniform interface. Stork is considered one of the very first examples of 'data-aware scheduling' and has been very actively used in many e-Science application areas, including: coastal hazard prediction, reservoir uncertainty analysis, digital sky imaging, educational video processing, numerical relativity and multi-scale computational fluid dynamics resulting in breakthrough research [5][6][7][8][9][10][38][39][40].…”

Section: Stork Data Schedulermentioning

confidence: 99%

Stork data scheduler: mitigating the data bottleneck in e-Science

Kosar

Balman

Yildirim

et al. 2011

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

In this paper, we present the Stork data scheduler as a solution for mitigating the data bottleneck in e-Science and data-intensive scientific discovery. Stork focuses on planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked inputs/outputs for petascale distributed e-Science applications. Unlike existing approaches, Stork treats data resources and the tasks related to data access and movement as first-class entities just like computational resources and compute tasks, and not simply the side-effect of computation. Stork provides unique features such as aggregation of data transfer jobs considering their source and destination addresses, and an application-level throughput estimation and optimization service. We describe how these two features are implemented in Stork and their effects on end-to-end data transfer performance.

show abstract

Data placement in widely distributed environments

Cited by 12 publications

References 17 publications

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

Task And Data Allocation Strategies for Big Data Workflows

Stork data scheduler: mitigating the data bottleneck in e-Science

Contact Info

Product

Resources

About