Dynamic Protocol Tuning Algorithms for High Performance Data Transfers

Abstract

Obtaining optimal data transfer performance is of utmost importance to today's data-intensive distributed applications and wide-area data replication services.Doing so necessitates effectively utilizing available network bandwidth and resources, yet in practice transfers seldom reach the levels of utilization they potentially could. Tuning protocol parameters such as pipelining, parallelism, and concurrency can significantly increase utilization and performance, however determining the best settings for these parameters is a difficult problem, as network conditions can vary greatly between sites and over time. Nevertheless, it is an important problem, since poor tuning can cause either under-or over-utilization of network resources and thus degrade transfer performance. In this paper, we present three algorithms for application-level tuning of different protocol parameters for maximizing transfer throughput in wide-area networks.Our algorithms dynamically tune the number of parallel data streams per file (for large file optimization), the level of control channel pipelining (for small file optimization), and the number of concurrent file transfers to increase I/O throughput (a technique useful for all types of files). The proposed heuristic algorithms improve the transfer throughput up to 10x compared to the baseline and 7x compared to the state of the art solutions.

Energy-aware data transfer algorithms

“…We compared the performance of our energy-aware MinE and HTEE algorithms with energy-agnostic Single Chunk (SC) and Pro-active Multi Chunk (ProMC) algorithms [10] as well as the popular cloud-hosted data transfer service Globus Online (GO) [6] and the standard Globus GridFTP client (Globus-url-copy). SC and GO algorithms employ divide and transfer approach to transfer a dataset with mixed sized files.…”

“…Thus, we let the user to be able to decide the maximum acceptable number of concurrent channels for a data transfer. We have chosen concurrency level to tune the throughput, since in our previous work [5,10] we have observed that concurrency is the most influential transfer parameter for all file sizes in most test environments. Even though parallelism also creates multiple channels similar to concurrency, allotting channels to multiple file transfer instead of a single one yields higher disk IO throughput which qualifies concurrency to be the most effective parameter on transfer throughput.…”

“…PCP [47] algorithm is proposed to find optimal values for transfer parameters such as pipelining, concurrency and parallelism. In another study [10], we presented four application level dynamic data transfer algorithms that tune the transfer parameters heuristically for reaching optimum network throughput but we did not consider the power consumption of these algorithms. In [4], we analyzed how each data transfer parameter may affect both the throughput and the power consumption in the end-toend data transfers.…”

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

“…HARP combines three approaches of application-level data transfer tuning and optimization: i) heuristics; (ii) real-time probing; and (iii) historical data analysis. Heuristic algorithms [19], [26] compute transfer parameters through calculations on the dataset and network metrics. For example, the value of pipelining is calculated by dividing the bandwidthdelay-product (BDP) to the average file size so that it will return large values for small files and small values for large files which aligns with the purpose of pipelining [27].…”

“…Optimal values for these parameters depend on the many factors described above. In our previous work, we have developed heuristicbased dynamic optimization algorithms [19] to determine the best combination of these parameters by using network and dataset characteristics (i.e., bandwidth, round-trip-time, and average file size).…”