DataStager: scalable data staging services for petascale applications

Abbasi, Hasan; Wolf, Matthew; Eisenhauer, Greg; Klasky, Scott; Schwan, Karsten; Zheng, Fang

doi:10.1007/s10586-010-0135-6

Cited by 123 publications

(109 citation statements)

References 29 publications

Supporting

Mentioning

107

Contrasting

Order By: Relevance

“…The IBM Blue Gene series of supercomputers (Yu et al, 2006) uses independent I/O nodes in their system to handle I/O requests, which are generated in computer nodes and forwarded to I/O nodes. DataStager, designed by Abbasi et al (2009), is a data staging service that provide asynchronous data extraction for ADIOS. ADIOS provides a simple function and an external XML file to configure the data structure and I/O methods.…”

Section: Related Workmentioning

confidence: 99%

A fast input/output library for high-resolution climate models

Wang

Yang

et al. 2014

Geosci. Model Dev.

View full text Add to dashboard Cite

Abstract. We describe the design and implementation of climate fast input/output (CFIO), a fast input/output (I/O) library for high-resolution climate models. CFIO provides a simple method for modelers to overlap the I/O phase with the computing phase automatically, so as to shorten the running time of numerical simulations. To minimize the code modifications required for porting, CFIO provides similar interfaces and features to parallel Network Common Data Form (PnetCDF), which is one of the most widely used I/O libraries in climate models. We deployed CFIO in three highresolution climate models, including two ocean models (POP and LICOM) and one sea ice model (CICE). The experimental results show that CFIO improves the performance of climate models significantly versus the original serial I/O approach. When running with CFIO at 0.1 • resolution with about 1000 CPU cores, we managed to reduce the running time by factors of 7.9, 4.6 and 2.0 for POP, CICE, and LI-COM, respectively. We also compared the performance of CFIO against two existing libraries, PnetCDF and parallel I/O (PIO), in different scenarios. For scenarios with both data output and computations, CFIO decreases the I/O overhead compared to PnetCDF and PIO.

show abstract

Section: Related Workmentioning

confidence: 99%

A fast input/output library for high-resolution climate models

Wang

Yang

et al. 2014

Geosci. Model Dev.

View full text Add to dashboard Cite

show abstract

“…If there is no dependence between data requested in the I/O call and the execution following it, I/O time could be hidden behind the computation time. For data-intensive scientific applications, researchers developed in-situ [10] execution or DataStager [9], where the input/output data can be asynchronously loaded/unloaded into/out of compute nodes or a staging area. Applying data asynchronous access usually requires programmers' or users' involvements.…”

Section: A Reducing Disk Latencymentioning

confidence: 99%

LiU: Hiding Disk Access Latency for HPC Applications with a New SSD-Enabled Data Layout

Huang¹,

Zhang

Shi

et al. 2013

2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems

View full text Add to dashboard Cite

Abstract-Unlike in the consumer electronics and personal computing areas, in the HPC environment hard disks can hardly be replaced by SSDs. The reasons include hard disk's large capacity, very low price, and decent peak throughput. However, when latency dominates the I/O performance (e.g., when accessing random data), the hard disk's performance can be compromised. If the issue of high latency could be effectively solved, the HPC community would enjoy a large, affordable and fast storage without having to replace disks completely with expensive SSDs.In this paper, we propose an almost latency-free hard-diskdominated storage system called LiU for HPC. The key technique is leveraging limited amount of SSD storage for its low-latency access, and changing data layout in a hybrid storage hierarchy with low-latency SSD at the top and high-latency hard disk at the bottom. If a segment of data would be randomly accessed, we lift its top part (the head) up in the hierarchy to the SSD and leave the remaining part (the body) untouched on the disk. As a result, the latency of accessing this whole segment can be removed because access latency of the body can be hidden by the access time of the head on the SSD. Combined with the effect of prefetching a large segment, LiU (Lift it Up) can effectively remove disk access latency so disk's high peak throughput can now be fully exploited for data-intensive HPC applications.We have implemented a prototype of LiU in the PVFS parallel file system and evaluated it with representative MPI-IO microbenchmarks, including mpi-io-test, mpi-tile-io, and ior-mpi-io, and one macro-benchmark BTIO. Our experimental results show that LiU can effectively improve the I/O performance for HPC applications, with the throughput improvement ratio up to 5.8. Furthermore, LiU can bring much more benefits to sequential-I/O MPI applications when the applications are interfered by other workloads. For example, LiU improves the I/O throughput of mpi-io-test, which is under interference, by 1.1-3.4 times, while improving the same workload without interference by 15%.

show abstract

“…However, ADIOS does support several staging methods (e.g., DataStager [39] and DataSpaces [40]), which provide in memory data to the application as an external component. Interval-IO differs from ADIOS most significantly in its support of the MPI-IO interface, which, as noted above, is one of the key goals of this research.…”

Section: Related Workmentioning

confidence: 99%