ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Byna, Suren; Breitenfeld, Scot; Dong, Bo; Koziol, Quincey; Pourmal, Elena; Robinson, Dana; Soumagne, Jérôme; Tang, Houjun; Vishwanath, Venkatram; Warren, Richard

doi:10.1007/s11390-020-9822-9

Cited by 38 publications

(17 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, a recent pilot study in computational fluid dynamics showed that it could be more than 200 times faster than the same workload on an optimized number of cores on the NETL's supercomputer JOULE 2.0 [Rocki et al, 2020]. Similar scaling performance has been reported on other exascale computing clusters involving hundreds of GPU's [Byna et al, 2020].…”

Section: Model Scalibilitymentioning

confidence: 70%

Applications of physics-informed scientific machine learning in subsurface science: A survey

Sun¹,

Yoon²,

Shih³

et al. 2021

Preprint

View full text Add to dashboard Cite

Geosystems are geological formations altered by humans activities such as fossil energy exploration, waste disposal, geologic carbon sequestration, and renewable energy generation. Geosystems also represent a critical link in the global water-energy nexus, providing both the source and buffering mechanisms for enabling societal adaptation to climate variability and change. The responsible use and exploration of geosystems are thus critical to the geosystem governance, which in turn depends on the efficient monitoring, risk assessment, and decision support tools for practical implementation. Large-scale, physics-based models have long been developed and used for geosystem management by incorporating geological domain knowledge such as stratigraphy, governing equations of flow and mass transport in porous media, geological and initial/boundary constraints, and field observations. Spatial heterogeneities and the multiscale nature of geological formations, however, pose significant challenges to the conventional numerical models, especially when used in a simulation-based optimization framework for decision support. Fast advances in machine learning (ML) algorithms and novel sensing technologies in recent years have presented new opportunities for the subsurface research community to improve the efficacy and transparency of geosystem governance. Although recent studies have shown the great promise of scientific ML (SciML) models, questions remain on how to best leverage ML in the management of geosystems, which are typified by multiscality, high-dimensionality, and data resolution inhomogeneity. This survey will provide a systematic review of the recent development and applications of domain-aware SciML in geosystem researches, with an emphasis on how the accuracy, interpretability, scalability, defensibility, and generalization skill of ML approaches can be improved to better serve the geoscientific community.

show abstract

Section: Model Scalibilitymentioning

confidence: 70%

Applications of physics-informed scientific machine learning in subsurface science: A survey

Sun¹,

Yoon²,

Shih³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Hierarchical Data Format version 5 (HDF5) [24], [25] is the most popular I/O middleware and file format used at NERSC. HDF5 provides a data model, library, and file format along with a rich programming interface for storing and managing data.…”

Section: E Hdf5 Features and Improvementsmentioning

confidence: 99%

Cross-facility science with the Superfacility Project at LBNL

Enders

Bard

Snavely

et al. 2020

2020 IEEE/ACM 2nd Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing (XLOOP)

Self Cite

View full text Add to dashboard Cite

As data sets from DOE user science facilities grow in both size and complexity there is an urgent need for new capabilities to transfer, analyze and manage the data underlying scientific discoveries. LBNL's Superfacility project brings together experimental and observational research instruments with computational and network facilities at the National Energy Research Scientific Computing Center (NERSC) and the Energy Sciences Network (ESnet) with the goal of enabling user science.Here, we report on recent innovations in the Superfacility project, including advanced data management, API-based automation, real-time interactive user interfaces, and supported infrastructure for "edge" services.

show abstract

“…HDF5 has been used heavily at supercomputing facilities for storing, reading, and querying scientific datasets [16], [17]. This is because HDF5 has specific designs and performance optimizations for popular parallel file systems such as Lustre [18], [19]. Moreover, HDF5 also provides users dynamically loaded filters [20] such as lossless and lossy compression [21], which can automatically store and query data in compressed formats.…”

Section: Introductionmentioning

confidence: 99%

Improving Prediction-Based Lossy Compression Dramatically via Ratio-Quality Modeling

Jin¹,

Di²,

Tian³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Error-bounded lossy compression is one of the most effective techniques for reducing scientific data sizes. However, the traditional trial-and-error approach used to configure lossy compressors for finding the optimal trade-off between reconstructed data quality and compression ratio is prohibitively expensive. To resolve this issue, we develop a general-purpose analytical ratio-quality model based on the prediction-based lossy compression framework, which can effectively foresee the reduced data quality and compression ratio, as well as the impact of lossy compressed data on post-hoc analysis quality. Our analytical model significantly improves the prediction-based lossy compression in three use-cases: (1) optimization of predictor by selecting the best-fit predictor; (2) memory compression with a target ratio; and (3) in-situ compression optimization by fine-grained tuning error-bounds for various data partitions. We evaluate our analytical model on 10 scientific datasets, demonstrating its high accuracy (93.47% accuracy on average) and low computational cost (up to 18.7× lower than the trialand-error approach) for estimating the compression ratio and the impact of lossy compression on post-hoc analysis quality. We also verify the high efficiency of our ratio-quality model using different applications across the three use-cases. In addition, our experiment demonstrates that our modeling based approach reduces the time to store the 3D RTM data with HDF5 by up to 3.4× with 128 CPU cores over the traditional solution.

show abstract

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Cited by 38 publications

References 10 publications

Applications of physics-informed scientific machine learning in subsurface science: A survey

Applications of physics-informed scientific machine learning in subsurface science: A survey

Cross-facility science with the Superfacility Project at LBNL

Improving Prediction-Based Lossy Compression Dramatically via Ratio-Quality Modeling

Contact Info

Product

Resources

About