Interfacing HDF5 with a scalable object‐centric storage system on hierarchical storage

Mu, Jingqing; Soumagne, Jérôme; Byna, Suren; Koziol, Quincey; Tang, Houjun; Warren, Richard

doi:10.1002/cpe.5715

Cited by 10 publications

(3 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mehta et al [31] develop a new HDF5 plugin in order to use the parallel file system to convert the single-file layout into a data layout that is optimized and stores data in a unique way that enables semantic post-processing on data. Mu et al [32], design a storage interface based on data containers that provides data chunking and takes advantage of multiple storage tiers.…”

Section: Related Workmentioning

confidence: 99%

Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization

Wan,

Huebl,

et al. 2021

Preprint

View full text Add to dashboard Cite

The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. We show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80%.

show abstract

Section: Related Workmentioning

confidence: 99%

Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization

Wan,

Huebl,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Three papers address file systems and networking layers. Mu et al present a means to take advantage of the Proactive Data Containers (PDC) object storage system through HDF5, enabling implicit and asynchronous data movement to various storage tiers with little to no code modification. The Virtual Object Layer (VOL) connector introduced in this work shows a significant performance gain over native HDF5 as file system accesses are handled by the PDC service rather than by the user application.…”

Section: Themes Of This Special Issuementioning

confidence: 99%

Foreword to the Special Issue of the Cray User Group

Robinson

Thota

2020

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Important access libraries have started to address this problem by introducing a new layer just below its API that allows the introduction of new backends with more up-to-date assumptions while not requiring any changes to the application. For example, HDF5 has introduced the Virtual Object Layer (VOL) [4] that has enabled an ecosystem of plugins that map the HDF5 API not only to its traditional Virtual File Layer (effectively re-implementing an entire file system on top of file) but also to object-based storage backends, including Amazon S3 and Ceph/RADOS [5]. Another example is NetCDF4 which maps to HDF5.…”

Section: Introductionmentioning

confidence: 99%

Mapping Datasets to Object Storage System

et al. 2020

Self Cite

View full text Add to dashboard Cite

Access libraries such as ROOT[1] and HDF5[2] allow users to interact with datasets using high level abstractions, like coordinate systems and associated slicing operations. Unfortunately, the implementations of access libraries are based on outdated assumptions about storage systems interfaces and are generally unable to fully benefit from modern fast storage devices. For example, access libraries often implement buffering and data layout that assume that large, single-threaded sequential access patterns are causing less overall latency than small parallel random access: while this is true for spinning media, it is not true for flash media. The situation is getting worse with rapidly evolving storage devices such as non-volatile memory and ever larger datasets. This project explores distributed dataset mapping infrastructures that can integrate and scale out existing access libraries using Ceph’s extensible object model, avoiding re-implementation or even modifications of these access libraries as much as possible. These programmable storage extensions coupled with our distributed dataset mapping techniques enable: 1) access library operations to be offloaded to storage system servers, 2) the independent evolution of access libraries and storage systems and 3) fully leveraging of the existing load balancing, elasticity, and failure management of distributed storage systems like Ceph. They also create more opportunities to conduct storage server-local optimizations specific to storage servers. For example, storage servers might include local key/value stores combined with chunk stores that require different optimizations than a local file system. As storage servers evolve to support new storage devices like non-volatile memory, these server-local optimizations can be implemented while minimizing disruptions to applications. We will report progress on the means by which distributed dataset mapping can be abstracted over particular access libraries, including access libraries for ROOT data, and how we address some of the challenges revolving around data partitioning and composability of access operations.

show abstract

Interfacing HDF5 with a scalable object‐centric storage system on hierarchical storage

Cited by 10 publications

References 28 publications

Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization

Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization

Foreword to the Special Issue of the Cray User Group

Mapping Datasets to Object Storage System

Contact Info

Product

Resources

About