Prediction of Optimal Parallelism Level in Wide Area Data Transfers

Yildirim, Esma; Yin, Dengpan; Kosar, Tevfik

doi:10.1109/tpds.2011.228

Cited by 49 publications

(25 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bulk data transfers using parallel TCP connections have been studied [42][43][44][45][46][47][48]. GridFTP [49] supports parallel and striped transfers by partitioning a file into a number of pieces to fully utilize the bandwidth.…”

Section: Related Workmentioning

confidence: 99%

“…These challenges have been subject of active research, and solutions addressing different aspects of WAN migrations have been proposed [8] [9][10] [11] [12] [13] [14][15] [16] [17] [18] [19] [20]. Improvements to the VM migration process could be also be achieved by, for example, applying bulk transfer techniques (e.g., parallel TCP [42]- [48], bulk UDP transfer [51] [52], and peer-to-peer mechanisms [53]). However, integrating such techniques to software controlling VMs (hypervisors), would make a solution difficult to maintain due to the existence of different hypervisors and versions of a hypervisor.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reducing the Migration Times of Multiple VMs on WANs Using a Feedback Controller

Kang

Tsugawa

Fortes

et al. 2013

2013 IEEE International Symposium on Parallel &Amp; Distributed Processing, Workshops and PHD Forum

View full text Add to dashboard Cite

Virtual machine (VM) migration is affected by network latency and throughput, which are highly fluctuating and unpredictable in wide-area networks (WANs). Hence, it is difficult to statically minimize the time required to transfer a large number of VMs across WAN. The goal of this work is to migrate as many VMs as possible during a given period of time and it is motivated by disaster recovery scenarios. One approach is to migrate a large number of VMs in parallel, but this leads to long migration times of each individual VM. Long migration times are problematic in catastrophic circumstances where resources are limited and can fail within a short period of time. Thus, it is important to shorten both the total time required to migrate multiple VMs and the migration time of individual VMs. Due to network performance fluctuations, the optimal number of parallel migrations changes over time. This work proposes a feedback-based controller that adapts to the number of parallel VM migrations in response to changes in a WAN. The controller implements an algorithm inspired by the TCP congestion avoidance algorithm in order to regulate the number of VMs in transit depending on the network conditions. Experiments using a prototype controller confirm that it is possible to control the migration of a set of VMs shortening both the total and individual migration times. The experiments show that the controller shortens the individual migration time by up to 5.7 fold compared to that of the static VM migrations where the number of parallel migrations does not change until all migrations are completed. The contributions of this work are 1) introducing migration strategies for multiple VMs on WANs and 2) proposing a hypervisor-independent controller that adapts to network bandwidth fluctuations in disaster scenarios.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Reducing the Migration Times of Multiple VMs on WANs Using a Feedback Controller

Kang

Tsugawa

Fortes

et al. 2013

2013 IEEE International Symposium on Parallel &Amp; Distributed Processing, Workshops and PHD Forum

View full text Add to dashboard Cite

show abstract

“…the performance degradation due to the overhead of opening too many parallel streams). In our previous work, we have developed two highly-accurate models based on Full Second-order [17] and Partial C-order [16]. These models would require as few as three sampling points in the best case to provide very accurate predictions, but in the worst case they could require up to six or seven sampling points for accurate results.…”

Section: Related Workmentioning

confidence: 99%

“…Parallel streams achieve high throughput by mimicking the behavior of individual streams and get a higher share of the available bandwidth [7][8][9][10][11][12][13][14][15]. In our previous study, we presented two new theoretical models (Partial C-order and Full Second-order) that are used to predict the behavior of parallel streams and discussed their assumptions in application [16,17]. We also applied those models and measured their accuracy against actual GridFTP [18] transfers with the improvements we have made [19].…”

Section: Introductionmentioning

confidence: 99%

A Highly-Accurate and Low-Overhead Prediction Model for Transfer Throughput Optimization

Kim

Yildirim

Kosar

2012

2012 SC Companion: High Performance Computing, Networking Storage and Analysis

Self Cite

View full text Add to dashboard Cite

An important bottleneck for data-intensive scalable computing systems is efficient utilization of the network links that connect the collaborating institutions with their remote partners, data sources, and computational sites. To alleviate this bottleneck, we propose an application-layer throughput optimization model based on parallel stream number prediction. This new model extends our two previous models (Partial C-order and Full Second-order) to achieve higher accuracy and lower overhead predictions. Our new model, called Full C-order, outperforms both of our previous models as well as the most relevant model by others, the Partial Second-order, in terms of both accuracy and efficiency. We test and compare these four models on emulated testbeds and on production environments using a wide variety of data set sizes, RTT, and bandwidth combinations. Our comprehensive experiments confirm the superiority of our new model to the other three models.

show abstract

“…The detailed derivation of this equation and variables as well as the selection strategy of the data points (n 1 , n 2 , n 3 ) are explained in our previous work [41].…”

Section: Throughput Optimizationmentioning

confidence: 99%

Stork data scheduler: mitigating the data bottleneck in e-Science

Kosar

Balman

Yildirim

et al. 2011

Phil. Trans. R. Soc. A.

Self Cite

View full text Add to dashboard Cite

In this paper, we present the Stork data scheduler as a solution for mitigating the data bottleneck in e-Science and data-intensive scientific discovery. Stork focuses on planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked inputs/outputs for petascale distributed e-Science applications. Unlike existing approaches, Stork treats data resources and the tasks related to data access and movement as first-class entities just like computational resources and compute tasks, and not simply the side-effect of computation. Stork provides unique features such as aggregation of data transfer jobs considering their source and destination addresses, and an application-level throughput estimation and optimization service. We describe how these two features are implemented in Stork and their effects on end-to-end data transfer performance.

show abstract

Prediction of Optimal Parallelism Level in Wide Area Data Transfers

Cited by 49 publications

References 7 publications

Reducing the Migration Times of Multiple VMs on WANs Using a Feedback Controller

Reducing the Migration Times of Multiple VMs on WANs Using a Feedback Controller

A Highly-Accurate and Low-Overhead Prediction Model for Transfer Throughput Optimization

Stork data scheduler: mitigating the data bottleneck in e-Science

Contact Info

Product

Resources

About