Hysteresis-based optimization of data transfer throughput

Nine, S Q Zulkar; Guner, Kemal; Kosar, Tevfik

doi:10.1145/2832099.2832104

Cited by 11 publications

(11 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A substantial body of related work in the literature focuses on prediction of network throughput [18,11,7,17,1,13,2]. However, many authors present only short-term forecasts.…”

Section: Related Workmentioning

confidence: 99%

Forecasting Network Throughput of Remote Data Access in Computing Grids

Begy

Barisits

Lassnig

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Computing grids are key enablers of computational science. Researchers from many fields (High Energy Physics, Bioinformatics, Climatology, etc.) employ grids for execution of distributed computational jobs. These computing workloads are typically data-intensive. The current state of the art approach for data access in grids is data placement: a job is scheduled to run at a specific data center, and its execution commences only once the complete input data has been transferred there. An alternative approach is remote data access: a job may stream the input data directly from arbitrary storage elements. Remote data access brings two innovative benefits: (1) the jobs can be executed asynchronously with respect to the data transfer; (2) when combined with data placement on the policy level, it can aid in the optimization of the network load, since these two data access methodologies partially exhibit nonoverlapping bottlenecks. However, in order to employ this technique systematically, the properties of its network throughput need to be studied carefully. This paper presents experimentally identified parameters of remote data access throughput, statistically tested formalization of these parameters and a derived throughput forecasting model. The model is applicable to large computing workloads, robust with respect to arbitrary dynamic changes in the grid infrastructure and exhibits a long-term prediction horizon. Its purpose is to assist various stakeholders of the grid in decision-making related to data access patterns. This work is based on measurements taken on the Worldwide LHC Computing Grid at CERN.Computing grids [5] are collections of geographically sparse data centers. Employing a wide range of heterogenous hardware, these participating facilities work together to reach a common goal in a coordinated manner. Grids store vast amounts of scientific data, and numerous users run computational jobs to

show abstract

“…A substantial body of related work in the literature focuses on prediction of network throughput [18,11,7,17,1,13,2]. However, many authors present only short-term forecasts.…”

Section: Related Workmentioning

confidence: 99%

Forecasting Network Throughput of Remote Data Access in Computing Grids

Begy

Barisits

Lassnig

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Finally, historical data methods [29], [30] model data transfer throughput based on dataset, network, and protocol metrics. In order to capture change in the network load, they either rely on recent historical data [30] or run sample transfers [29]. HARP runs sample transfers similar to probing based solutions, but [29] require offline analysis and can work well only for networks for which they are trained, HARP requires no prior data analysis and can be applied to different networks with the help of extensible similarity detection algorithm.…”

Section: Overview Of Harpmentioning

confidence: 99%

“…In order to capture change in the network load, they either rely on recent historical data [30] or run sample transfers [29]. HARP runs sample transfers similar to probing based solutions, but [29] require offline analysis and can work well only for networks for which they are trained, HARP requires no prior data analysis and can be applied to different networks with the help of extensible similarity detection algorithm. HARP is composed of two main modules which are Scheduler and Optimizer as shown in Figure 2.…”

Section: Overview Of Harpmentioning

confidence: 99%

“…We compared HARP against heuristic (Globus Online [26], Single Chunk, and ProActive Multi-Chunk [19]), probing based [28], and hysteresis based [29] algorithms. Globus Online (GO) separates the dataset into chunks based on file size and uses predefined values for protocol parameters for each chunk's transfer.…”

Section: Experimental Analysismentioning

confidence: 99%

“…In our earlier work we have proposed heuristic algorithms [19] to determine the best parameter combination by using network and dataset characteristics (i.e bandwidth, round-trip-time, and average file size etc.). Nine et al developed ANN+OT [29] which uses historical data to derive model that relates transfer metrics to transfer throughput. It then runs real time probing in order to capture current network status.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

Arslan

Kosar

2018

IEEE Trans. Parallel Distrib. Syst.

Self Cite

View full text Add to dashboard Cite

Data-intensive scientific and commercial applications increasingly require frequent movement of large datasets from one site to the other(s). Despite growing network capacities, these data movements rarely achieve the promised data transfer rates of the underlying physical network due to poorly tuned data transfer protocols. Accurately and efficiently tuning the data transfer protocol parameters in a dynamically changing network environment is a major challenge and remains as an open research problem. In this paper, we present predictive end-to-end data transfer optimization algorithms based on historical data analysis and real-time background traffic probing, dubbed HARP. Most of the previous work in this area are solely based on real time network probing which results either in an excessive sampling overhead or fails to accurately predict the optimal transfer parameters. Combining historical data analysis with real time sampling enables our algorithms to tune the application level data transfer parameters accurately and efficiently to achieve close-to-optimal end-to-end data transfer throughput with very low overhead. Our experimental analysis over a variety of network settings shows that HARP outperforms existing solutions by up to 50% in terms of the achieved throughput.

show abstract