An empirical study of I/O separation for burst buffers in HPC systems

Koo, Donghun; Lee, John J.; Liu, Jialin; Byun, Eun-Kyu; Kwak, Jae-Hyuck; Lockwood, Glenn K.; Hwang, Soonwook; Antypas, Katie; Wu, Kesheng; Eom, Hyeonsang

doi:10.1016/j.jpdc.2020.10.007

Cited by 18 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This becomes especially interesting for cases when the checkpoint data is not needed for analyses. To improve the total checkpoint performance, without changing to a faster checkpoint method, one could leverage burst buffer storage [17], [23], [22], if available on the cluster.…”

Section: Discussionmentioning

confidence: 99%

Towards Zero-Waste Recovery and Zero-Overhead Checkpointing in Ensemble Data Assimilation

Keller

Kestelman

Bautista-Gomez

2021

2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)

View full text Add to dashboard Cite

Ensemble data assimilation is a powerful tool for increasing the accuracy of climatological states. It is based on combining observations with the results from numerical model simulations. The method comprises two steps, (1) the propagation, where the ensemble states are advanced by the numerical model and (2) the analysis, where the model states are corrected with observations. One bottleneck in ensemble data assimilation is circulating the ensemble states between the two steps. Often, the states are circulated using files. This article presents an extended implementation of Melissa-DA, an in-memory ensemble data assimilation framework, allowing zero-overhead checkpointing and recovery with few or zero recomputation. We hide the checkpoint creation using dedicated threads and MPI processes. We benchmark our implementation with up to 512 members simulating the Lorenz96 model using 10 9 gridpoints. We utilize up to 8 K processes and 8 TB of checkpoint data per cycle and reach a peak performance of 52 teraFLOPS.

show abstract

Section: Discussionmentioning

confidence: 99%

Towards Zero-Waste Recovery and Zero-Overhead Checkpointing in Ensemble Data Assimilation

Keller

Kestelman

Bautista-Gomez

2021

2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)

View full text Add to dashboard Cite

show abstract

“…Although SSDs do not have such seek penalties, they do require periodic garbage collection as erase blocks become sparse, and this results in a distinct loss of bandwidth once an SSD becomes sufficiently "dirty" [15]. I/O separation has been shown to mitigate this age-related performance loss on parallel file systems [16], but such techniques require hardware support implemented in the file system, and Lustre does not yet support any such capabilities.…”

Section: B Performance Over Timementioning

confidence: 99%

Architecture and performance of Perlmutter's 35 PB ClusterStor E1000 all‐flash file system

Lockwood,

Chiusole,

Gerhardt

et al. 2024

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SummaryNERSC's newest system, Perlmutter, features a 35 PB all‐flash Lustre file system built on HPE Cray ClusterStor E1000. We present its architecture, early performance figures, and performance considerations unique to this architecture. We demonstrate the performance of E1000 OSSes through low‐level Lustre tests that achieve over 90% of the theoretical bandwidth of the SSDs at the OST and LNet levels. We also show end‐to‐end performance for both traditional dimensions of I/O performance (peak bulk‐synchronous bandwidth) and nonoptimal workloads endemic to production computing (small, incoherent I/Os at random offsets) and compare them to NERSC's previous system, Cori, to illustrate that Perlmutter achieves the performance of a burst buffer and the resilience of a scratch file system. Finally, we discuss performance considerations unique to all‐flash Lustre and present ways in which users and HPC facilities can adjust their I/O patterns and operations to make optimal use of such architectures.

show abstract

“…The node runs simulations using Simple Linux Utility for Resource Management (SLURM). SLURM is a cluster workload manager for the Linux operating system [16]. This facility implements three main functions.…”

Section: A Material's Configurationsmentioning

confidence: 99%

An Investigation of Computation Time Based on Domain Size in WRF Model

Nafiisyanti¹,

Fathrio²

2022

International Journal on Advanced Science, Engineering and Information Technology

View full text Add to dashboard Cite

Estimating WRF model computation time is necessary because of the need for domain expansion for weather prediction. However, the optimum computation time is not simply gained by enlarging the domain and adding processor numbers. Thus, an investigation was carried out to determine the correlation between computation time on domain size, number of grids, and number of processors used to run the WRF model. This study uses a collection of computation time as the data input from running the WRF model with two domain group ratios, 2: 1 and 1: 1, and various processors. Negative Exponential Function (NEF) and Power Function (PF) as exponential decay functions are evaluated to represent the curve formed from the computation time against the number of processors in one domain case. This study also evaluates the speed up and efficiency of the use of processor numbers against the tested domains. NEF represents the decrease in computation time curve in a domain case better than PF since this function has a steeper slope, better initial value, and k constant that keeps the computation time falling below 0. The computation time can be optimally saved by adding approximately eight processors at the same domain ratio with four times larger grid size and the same amount of grid number. Investigations for other domain ratios need to be carried out to determine the characteristics of computation time on the number of processors, grid size, and domain size.

show abstract

An empirical study of I/O separation for burst buffers in HPC systems

Cited by 18 publications

References 17 publications

Towards Zero-Waste Recovery and Zero-Overhead Checkpointing in Ensemble Data Assimilation

Towards Zero-Waste Recovery and Zero-Overhead Checkpointing in Ensemble Data Assimilation

Architecture and performance of Perlmutter's 35 PB ClusterStor E1000 all‐flash file system

An Investigation of Computation Time Based on Domain Size in WRF Model

Contact Info

Product

Resources

About