An Anticipative Recursively Adjusting Mechanism for parallel file transfer in data grids

Abstract

SUMMARYData Grids enable the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for content needed by large-scale data-intensive applications such as high-energy physics, bioinformatics, and virtual astrophysical observatories. In Data Grids, co-allocation architectures were developed to enable parallel downloads of data sets from selected replica servers. As Internet is usually the underlying network of a grid, network bandwidth plays as the main factor affecting file transfers between clients and servers. In this paradigm, there are still some challenges that need to be solved, such as to reduce differences in finish times between selected replica servers, to avoid traffic congestion resulting from transferring the same blocks in different links among servers and clients, and to manage network performance variations among parallel transfers. In this paper, we propose the Anticipative Recursively Adjusting Mechanism (ARAM) scheme to adjust the workloads on selected replica servers and handle unpredictable variations in network performance by those servers. Our algorithm is based on using the finish rates for previously assigned transfers to anticipate the bandwidth status for the next section to adjust workloads, and to reduce file transfer times in grid environments. Our approach is useful in grid environments with unstable network link. It not only reduces idle time wasted waiting for the slowest server, but also decreases file transfer completion times.

Accessing medical image file with co-allocation HDFS in cloud

“…Previous research [11,13,22,24,25,27] shows that co-allocation can solve the data transfer problem in grid environments; these results are categorized: RAM [12,24,22], DAS, [27,13] and ARAM [28,23]. This concept is the foundation for the current study.…”

“…This allows the best downloading strategy. Earlier research [11,13,22] adopts the same co-allocation mechanism.…”

“…Current virtualization fault tolerance response plan, focusing on recovery failure, usually relies on a checkpoint/restart mechanism. However, in today's systems, node failures can often be predicted by detecting the deterioration of health conditions [22][23][24][25][26][27][28][29][30][31][32][33].…”