A data throughput prediction and optimization service for widely distributed many-task computing

Abstract

In this paper, we present the design and implementation of a network throughput prediction and optimization service for many-task computing in widely distributed environments. This service uses multiple parallel TCP streams to improve the end-to-end throughput of data transfers. A novel mathematical model is used to decide the number of parallel streams to achieve best performance. This model can predict the optimal number of parallel streams with as few as three prediction points. We implement this new service in the Stork data scheduler, where the prediction points can be obtained using Iperf and GridFTP samplings. Our results show that the prediction cost plus the optimized transfer time is much less than the unoptimized transfer time in most cases.

A Highly-Accurate and Low-Overhead Prediction Model for Transfer Throughput Optimization

“…the performance degradation due to the overhead of opening too many parallel streams). In our previous work, we have developed two highly-accurate models based on Full Second-order [17] and Partial C-order [16]. These models would require as few as three sampling points in the best case to provide very accurate predictions, but in the worst case they could require up to six or seven sampling points for accurate results.…”

“…These models lay the foundations of our current work for a highly-accurate and low-overhead prediction model for transfer throughput optimization. Partial C-order_1_2_4 Partial second-order_1_2 The existing prediction models (Partial Second-order [23], Full Second-order [17] and Partial C-order [16]) worked with as few as two or three sampling points, but choosing the best two or three sampling points was a major challenge in those models. If we randomly choose these two or three data points, the resulting approximation may be highly inaccurate (see Figure 1), since there is a high possibility that these random points may not reflect the characteristics of the actual throughput curve.…”

“…Parallel streams achieve high throughput by mimicking the behavior of individual streams and get a higher share of the available bandwidth [7][8][9][10][11][12][13][14][15]. In our previous study, we presented two new theoretical models (Partial C-order and Full Second-order) that are used to predict the behavior of parallel streams and discussed their assumptions in application [16,17]. We also applied those models and measured their accuracy against actual GridFTP [18] transfers with the improvements we have made [19].…”

Bulk Data Movement for Climate Dataset: Efficient Data Transfer Management with Dynamic Transfer Adjustment

“…The systems probe and measurements with external profilers are needed. Complex models are used to calculate the optimum number of multiple streams with the help of sample measurements in order to make a prediction [23,25,26]. Further, network conditions may change over time in the shared environments, and the estimated value might not reflect the most recent state of the system.…”

“…The proposed methodology operates without depending on any historical measurements and does not use external profiles for measurement. Instead of using predictive sampling as proposed in [17,25,26], we make use of the instant throughput information gathered from the actual data transfer operations that are currently active. The number of multiple streams is set dynamically in an adaptive manner by gradually increasing the number of concurrent connections up to an optimal point.…”