HARP: Predictive Transfer Optimization Based on Historical Analysis and Real-Time Probing

Arslan, Engin; Guner, Kemal; Kosar, Tevfik

doi:10.1109/sc.2016.24

“…A substantial body of related work in the literature focuses on prediction of network throughput [18,11,7,17,1,13,2]. However, many authors present only short-term forecasts.…”

Section: Related Workmentioning

confidence: 99%

“…if the amount of the overall traffic increases, the time to transfer an additional unit will increase shortly after). We have mined 4 different traffic time series from the logs recorded by the WLCG's distributed data management system Rucio [6] for the investigated period at the data center Australia-ATLAS: (1) intra-data center traffic in bytes; (2) inter-data center traffic in bytes; (3) sum of intra-and inter-data center traffic in bytes; (4) amount of intra-data center file accesses. It can be seen from the cross-correlation plots in Fig.…”

Section: Forecasting Modelmentioning

confidence: 99%

Forecasting network throughput of remote data access in computing grids

Begy

¹

,

Barisits

²

,

Lassnig

³

et al. 2020

Journal of Computational Science

View full text Add to dashboard Cite

Computing grids are key enablers of computational science. Researchers from many fields (High Energy Physics, Bioinformatics, Climatology, etc.) employ grids for execution of distributed computational jobs. These computing workloads are typically data-intensive. The current state of the art approach for data access in grids is data placement: a job is scheduled to run at a specific data center, and its execution commences only once the complete input data has been transferred there. An alternative approach is remote data access: a job may stream the input data directly from arbitrary storage elements. Remote data access brings two innovative benefits: (1) the jobs can be executed asynchronously with respect to the data transfer; (2) when combined with data placement on the policy level, it can aid in the optimization of the network load, since these two data access methodologies partially exhibit nonoverlapping bottlenecks. However, in order to employ this technique systematically, the properties of its network throughput need to be studied carefully. This paper presents experimentally identified parameters of remote data access throughput, statistically tested formalization of these parameters and a derived throughput forecasting model. The model is applicable to large computing workloads, robust with respect to arbitrary dynamic changes in the grid infrastructure and exhibits a long-term prediction horizon. Its purpose is to assist various stakeholders of the grid in decision-making related to data access patterns. This work is based on measurements taken on the Worldwide LHC Computing Grid at CERN.

show abstract

“…A substantial body of related work in the literature focuses on prediction of network throughput [18,11,7,17,1,13,2]. However, many authors present only short-term forecasts.…”

Section: Related Workmentioning

confidence: 99%

“…if the amount of the overall traffic increases, the time to transfer an additional unit will increase shortly after). We have mined 4 different traffic time series from the logs recorded by the WLCG's distributed data management system Rucio [6] for the investigated period at the data center Australia-ATLAS: (1) intra-data center traffic in bytes; (2) inter-data center traffic in bytes; (3) sum of intra-and inter-data center traffic in bytes; (4) amount of intra-data center file accesses. It can be seen from the cross-correlation plots in Fig.…”

Section: Parameters Of Remote Data Access Throughputmentioning

confidence: 99%

“…The job may commence its execution only after the completion of the following workflow: (1) the input replica is transferred from the relevant storage element at DC 2 to a storage element at DC 1 ; (2) the input replica is staged-in from the relevant storage element at DC 1 to the worker node. This simplistic approach has two major disadvantages: (1) the jobs are staying idle while waiting for the input data; (2) due to the limited infrastructure resources the distributed data management system handling the data placement may queue the transfers up to several days. An alternative approach is remote data access, which allows to directly stream the input replica from a local or remote storage element.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Forecasting Network Throughput of Remote Data Access in Computing Grids

Begy

¹

,

Barisits

²

,

Lassnig

³

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Computing grids are key enablers of computational science. Researchers from many fields (High Energy Physics, Bioinformatics, Climatology, etc.) employ grids for execution of distributed computational jobs. These computing workloads are typically data-intensive. The current state of the art approach for data access in grids is data placement: a job is scheduled to run at a specific data center, and its execution commences only once the complete input data has been transferred there. An alternative approach is remote data access: a job may stream the input data directly from arbitrary storage elements. Remote data access brings two innovative benefits: (1) the jobs can be executed asynchronously with respect to the data transfer; (2) when combined with data placement on the policy level, it can aid in the optimization of the network load, since these two data access methodologies partially exhibit nonoverlapping bottlenecks. However, in order to employ this technique systematically, the properties of its network throughput need to be studied carefully. This paper presents experimentally identified parameters of remote data access throughput, statistically tested formalization of these parameters and a derived throughput forecasting model. The model is applicable to large computing workloads, robust with respect to arbitrary dynamic changes in the grid infrastructure and exhibits a long-term prediction horizon. Its purpose is to assist various stakeholders of the grid in decision-making related to data access patterns. This work is based on measurements taken on the Worldwide LHC Computing Grid at CERN.Computing grids [5] are collections of geographically sparse data centers. Employing a wide range of heterogenous hardware, these participating facilities work together to reach a common goal in a coordinated manner. Grids store vast amounts of scientific data, and numerous users run computational jobs to

show abstract