A Highly-Accurate and Low-Overhead Prediction Model for Transfer Throughput Optimization

Abstract

An important bottleneck for data-intensive scalable computing systems is efficient utilization of the network links that connect the collaborating institutions with their remote partners, data sources, and computational sites. To alleviate this bottleneck, we propose an application-layer throughput optimization model based on parallel stream number prediction. This new model extends our two previous models (Partial C-order and Full Second-order) to achieve higher accuracy and lower overhead predictions. Our new model, called Full C-order, outperforms both of our previous models as well as the most relevant model by others, the Partial Second-order, in terms of both accuracy and efficiency. We test and compare these four models on emulated testbeds and on production environments using a wide variety of data set sizes, RTT, and bandwidth combinations. Our comprehensive experiments confirm the superiority of our new model to the other three models.

Tuning GridFTP Pipelining, Concurrency and Parallelism Based on Historical Data

“…In future work, the author can add more experiments to extend the historical database, based on different parameters such as RTT and bandwidth. Also, the author can combine the prediction model and the previous model [2] to provide a better solution finding optimal PCP values in the real network environments.…”

“…However, setting the optimal numbers for these parameters is a challenging problem, since poorly tuned parameters can cause underutilization of the network or they can overburden the network and degrade the performance due to increased packet loss, endsystem overhead, and other factors. In addition, the author's previous work [1], [2] sought to find a solution for an efficient big data transfer by utilizing the well-known protocol GridFTP with PCP (Pipelining-Concurrency-Parallelism). In this paper, the author proposes a prediction model based on historical data to find optimal values for pipelining, concurrency and parallelism and reports very promising experimental results.…”

Energy-saving Cross-layer Optimization of Big Data Transfer Based on Historical Log Analysis

“…There has been several efforts in Kasparan et al [17] analyzed how pipelining affects throughput in local area networks and high-speed downlink packet access networks. Kim et al [19] and Yildirim et al [32] considered the combined effect of parallelism, pipelining, and concurrency on end-toend data transfer throughput. Natarajan et al [28] showed that using a single stream for transferring independent web objects is very inefficient in high latency networks.…”

Dynamic Protocol Tuning Algorithms for High Performance Data Transfers

“…In our previous work [18], we proposed network-aware transfer optimization by automatically detecting bottlenecks and improving throughput via utilization of network and end-system parallelism. We developed three highly-accurate models [19,20,21] which would require as few as three sampling points to provide accurate predictions for the optimal parallel stream number. These models have proved to be more accurate than existing similar models [22,12] which lack in predicting the parallel stream number that gives the peak throughput.…”

“…However, setting the optimal levels for these parameters is a challenging problem, and poorly-tuned parameters can either cause underutilization of the network or overburden the network and degrade the performance due to increased packet loss, end-system overhead, and other factors. Among these parameters, pipelining specifically targets the problem of transferring a large numbers of small files [13,14,28]. In most control channelbased transfer protocols, an entire transfer must complete and be acknowledged before the next transfer command is sent by the client.…”