Lightweight I/O for Scientific Applications

Oldfield, Ron A.; Ward, Lee; Riesen, Rolf; Maccabe, A.B.; Widener, Patrick; Kordenbrock, Todd

doi:10.1109/clustr.2006.311853

Cited by 59 publications

(42 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, those semantics are not always required for storage operations. By removing all but the most essential services, a custom file system can give excellent performance since it does not pay the performance penalties of any services that are not required [26]. While this works well, it would still require customizations similar to the proposed work so that it can adapt to different application requirements.…”

Section: Related Workmentioning

confidence: 99%

“…For example, the Lightweight File Systems [26] project at Sandia Labs has stripped down POSIX semantics to a core of authentication and authorization affording layering of other semantics, like consistency, on an as-needed basis. Other file systems like the Serverless File System [3] have distributed metadata weakening the immediate consistency across the entire network of machines.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Advanced I/O for large-scale scientific applications.

Klasky

Schwan

Oldfield³

et al. 2010

Self Cite

View full text Add to dashboard Cite

As scientific simulations scale to use petascale machines and beyond, the data volumes generated pose a dual problem. First, with increasing machine sizes, the careful tuning of IO routines becomes more and more important to keep the time spent in IO acceptable. It is not uncommon, for instance, to have 20% of an application's runtime spent performing IO in a 'tuned' system. Careful management of the IO routines can move that to 5% or even less in some cases. Second, the data volumes are so large, on the order of 10s to 100s of TB, that trying to discover the scientifically valid contributions requires assistance at runtime to both organize and annotate the data. Waiting for offline processing is not feasible due both to the impact on the IO system and the time required. To reduce this load and improve the ability of scientists to use the large amounts of data being produced, new techniques for data management are required. First, there is a need for techniques for efficient movement of data from the 3 compute space to storage. These techniques should understand the underlying system infrastructure and adapt to changing system conditions. Technologies include aggregation networks, data staging nodes for a closer parity to the IO subsystem, and autonomic IO routines that can detect system bottlenecks and choose different approaches, such as splitting the output into multiple targets, staggering output processes. Such methods must be end-to-end, meaning that even with properly managed asynchronous techniques, it is still essential to properly manage the later synchronous interaction with the storage system to maintain acceptable performance. Second, for the data being generated, annotations and other metadata must be incorporated to help the scientist understand output data for the simulation run as a whole, to select data and data features without concern for what files or other storage technologies were employed. All of these features should be attained while maintaining a simple deployment for the science code and eliminating the need for allocation of additional computational resources.4 Acknowledgment

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Advanced I/O for large-scale scientific applications.

Klasky

Schwan

Oldfield³

et al. 2010

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, such server-directed I/O [7] leaves much more room for optimisation. For example, we let the HSM system reorder the files in the staging list such that files close to one another on the same tape volume will be staged together, reducing many random seeks into fewer sequential reads.…”

Section: Access Intervals (Days)mentioning

confidence: 99%

Optimising NGAS for the MWA Archive

Wicenec

Pallot

et al. 2013

Exp Astron

View full text Add to dashboard Cite

The Murchison Widefield Array (MWA) is a next-generation radio telescope, generating visibility data products continuously at about 400 MB/s. Efficiently managing and archiving this data is a challenge. The MWA Archive consists of dataflows and storage sub-systems distributed across three tiers. At its core is the open source software -the Next-Generation Archive System (NGAS) -that was initially developed in ESO. However, to meet the MWA data challenge, we have tailored and optimised NGAS to achieve high-throughput data ingestion, efficient data flow management, multi-tiered data storage and processing-aware data staging.

show abstract

“…I/O forwarding has also been used in Prost et al (2001), Oldfield et al (2006), Nisar et al (2008), Fu et al (2010), Docan et al (2010) and May (2001) to reduce the I/O impact on computing. The IBM Blue Gene series of supercomputers (Yu et al, 2006) uses independent I/O nodes in their system to handle I/O requests, which are generated in computer nodes and forwarded to I/O nodes.…”

Section: Related Workmentioning

confidence: 99%

A fast input/output library for high-resolution climate models

Wang

Yang

et al. 2014

Geosci. Model Dev.

View full text Add to dashboard Cite

Abstract. We describe the design and implementation of climate fast input/output (CFIO), a fast input/output (I/O) library for high-resolution climate models. CFIO provides a simple method for modelers to overlap the I/O phase with the computing phase automatically, so as to shorten the running time of numerical simulations. To minimize the code modifications required for porting, CFIO provides similar interfaces and features to parallel Network Common Data Form (PnetCDF), which is one of the most widely used I/O libraries in climate models. We deployed CFIO in three highresolution climate models, including two ocean models (POP and LICOM) and one sea ice model (CICE). The experimental results show that CFIO improves the performance of climate models significantly versus the original serial I/O approach. When running with CFIO at 0.1 • resolution with about 1000 CPU cores, we managed to reduce the running time by factors of 7.9, 4.6 and 2.0 for POP, CICE, and LI-COM, respectively. We also compared the performance of CFIO against two existing libraries, PnetCDF and parallel I/O (PIO), in different scenarios. For scenarios with both data output and computations, CFIO decreases the I/O overhead compared to PnetCDF and PIO.

show abstract

Lightweight I/O for Scientific Applications

Cited by 59 publications

References 18 publications

Advanced I/O for large-scale scientific applications.

Advanced I/O for large-scale scientific applications.

Optimising NGAS for the MWA Archive

A fast input/output library for high-resolution climate models

Contact Info

Product

Resources

About