Implementation of a dynamic adjustment strategy for parallel file transfer in co-allocation data grids

Yang, Chao-Tung; Wang, Shih-Yu; Chu, William Cheng‐Chung

doi:10.1007/s11227-009-0307-4

Cited by 13 publications

(12 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When we performed data transfers using the GridFTP protocol, we discovered that CPU and I/O statuses slightly affect data transfer performance. Their respective values in our Data Grid environment are 80, 10, and 10% [21].…”

Section: Our Replica Selection Cost Modelmentioning

confidence: 99%

RACAM: design and implementation of a recursively adjusting co‐allocation method with efficient replica selection in Data Grids

Yang

Chen

2010

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARYData Grids enable the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for addressing large-scale data-intensive scientific application needs in, for instance, high-energy physics, bioinformatics, and virtual astrophysical observatories. Data sets are replicated in Data Grids and distributed among multiple sites. Unfortunately, data sets of interest sometimes are significantly large in size, and may cause access efficiency overhead. A co-allocation architecture was developed in order to enable parallel downloading of data sets from multiple servers. Several co-allocation strategies have been coupled and used to exploit download rate by specifying among various client-server divides files into multiple blocks of equal sizes to link and address dynamic rate fluctuations. However, one major obstacle, the idle time of faster servers having to wait for the slowest server to deliver the final block, makes it important to reduce differences in finishing time among replica servers. In this paper, we propose a dynamic co-allocation method, called Recursively Adjusting Co-Allocation Method (RACAM), to improve the performance of parallel data file transfer. Our approach reduces the idle time spent waiting for the slowest server and decreases data transfer completion time. We also provide an effective scheme for reducing the cost of reassembling data blocks.

show abstract

Section: Our Replica Selection Cost Modelmentioning

confidence: 99%

RACAM: design and implementation of a recursively adjusting co‐allocation method with efficient replica selection in Data Grids

Yang

Chen

2010

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the case of downloading from dynamic servers, dynamic co-allocation technique is used to dynamically select source servers according to the current network situation and the server loads. The dynamic co-allocation problem has been addressed in many researches [7,8,[24][25][26][27][28][29][30][31][32]. All these researches focus on finding a perfect dynamic co-allocation algorithm that can achieve high download speed from multiple source servers in a dynamic changing environment where server may be dead or slow down over time.…”

Section: Related Workmentioning

confidence: 99%

“…The authors in [24][25][26][27][28] propose a recursive-adjustment co-allocation scheme, to reduce the idle time of faster servers waiting for the slowest server to finish the final blocks. The proposed scheme continuously adjusts the replica server's workload to affect its real-time network bandwidth.…”

Section: Related Workmentioning

confidence: 99%

Improving job scheduling performance with parallel access to replicas in Data Grid environment

et al. 2009

View full text Add to dashboard Cite

Data Grid has evolved to be the solution for data-intensive applications, such as High Energy Physics (HEP), astrophysics, and computational genomics. These applications usually have large input of data to be analyzed and these input data are widely replicated across Data Grid to improve the performance. The job scheduling performance on traditional computing jobs can be studied using queuing theory. However, with the addition of data transfer, the job scheduling performance is too complex to be modeled. In this research, we study the impact of data transfer on the performance of job scheduling in the Data Grid environment. We have proposed a parallel downloading system that supports replicating data fragments and parallel downloading of replicated data fragments, to improve the job scheduling performance. The performance of the parallel downloading system is compared with non-parallel downloading system, using three scheduling heuristics: Shortest Turnaround Time (STT), Least Relative Load (LRL) and Data Present (DP). Our simulation results show that the proposed parallel download approach greatly improves the Data Grid performance for all three scheduling algorithms, in terms of the geometric mean of job turnaround time. The advantage of parallel downloading system is most evident when the Data Grid has relatively low network bandwidth and relatively high computing power.

show abstract

“…Current virtualization fault tolerance response plan, focusing on recovery failure, usually relies on a checkpoint/restart mechanism. However, in today's systems, node failures can often be predicted by detecting the deterioration of health conditions [22][23][24][25][26][27][28][29][30][31][32][33].…”

Section: Introductionmentioning

confidence: 99%

Accessing medical image file with co-allocation HDFS in cloud

Yang

Shih

Chen

et al. 2015

Future Generation Computer Systems

Self Cite

View full text Add to dashboard Cite

Implementation of a dynamic adjustment strategy for parallel file transfer in co-allocation data grids

Cited by 13 publications

References 22 publications

RACAM: design and implementation of a recursively adjusting co‐allocation method with efficient replica selection in Data Grids

RACAM: design and implementation of a recursively adjusting co‐allocation method with efficient replica selection in Data Grids

Improving job scheduling performance with parallel access to replicas in Data Grid environment

Accessing medical image file with co-allocation HDFS in cloud

Contact Info

Product

Resources

About