A 1 PB/s file system to checkpoint three million MPI tasks

Rajachandrasekar, Raghunath; Moody, Adam; Mohror, Kathryn; Panda, Dhabaleswar K.

doi:10.1145/2462902.2462908

Cited by 16 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, with the increasing scale of applications, they need to write larger checkpoint data more frequently (Sato et al, 2014). This generates an enormous amount of write traffic to storage systems (Rajachandrasekar et al, 2013). Much work (Ali et al, 2009;Dongarra, 2010;Shalf et al, 2010;Bent et al, 2012;Lofstead et al, 2016) has shown that current HDD-based storage systems have been stretched to their limits in handling the tremendous amount of I/O.…”

Section: Introductionmentioning

confidence: 99%

ONFS: a hierarchical hybrid file system based on memory, SSD, and HDD for high performance computers

Liu

Yu³

et al. 2017

Frontiers Inf Technol Electronic Eng

View full text Add to dashboard Cite

With supercomputers developing towards exascale, the number of compute cores increases dramatically, making more complex and larger-scale applications possible. The input/output (I/O) requirements of large-scale applications, workflow applications, and their checkpointing include substantial bandwidth and an extremely low latency, posing a serious challenge to high performance computing (HPC) storage systems. Current hard disk drive (HDD) based underlying storage systems are becoming more and more incompetent to meet the requirements of next-generation exascale supercomputers. To rise to the challenge, we propose a hierarchical hybrid storage system, on-line and near-line file system (ONFS). It leverages dynamic random access memory (DRAM) and solid state drive (SSD) in compute nodes, and HDD in storage servers to build a three-level storage system in a unified namespace. It supports portable operating system interface (POSIX) semantics, and provides high bandwidth, low latency, and huge storage capacity. In this paper, we present the technical details on distributed metadata management, the strategy of memory borrow and return, data consistency, parallel access control, and mechanisms guiding downward and upward migration in ONFS. We implement an ONFS prototype on the TH-1A supercomputer, and conduct experiments to test its I/O performance and scalability. The results show that the bandwidths of single-thread and multi-thread 'read'/'write' are 6-fold and 5-fold better than HDD-based Lustre, respectively. The I/O bandwidth of data-intensive applications in ONFS can be 6.35 times that in Lustre.

show abstract

Section: Introductionmentioning

confidence: 99%

ONFS: a hierarchical hybrid file system based on memory, SSD, and HDD for high performance computers

Liu

Yu³

et al. 2017

Frontiers Inf Technol Electronic Eng

View full text Add to dashboard Cite

show abstract

“…My primary task is to improve the performance of UnifyFS in checkpoint restarts. For this purpose, I will delve into checkpointing [16] technology and fine-tune its generation, storage, and recovery processes [17] , aiming to enhance the reliability and efficiency of the entire system. Simultaneously, I plan to integrate the Veloc tool into UnifyFS, which will enhance the system's capabilities in data management and recovery, and offer more flexible and efficient solutions through data snapshots and version control.…”

Section: Discussionmentioning

confidence: 99%

Research on the integration of UnifyFS and Slurm

Bai,

Zhu,

Zhang

et al. 2024

Third International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2024)

View full text Add to dashboard Cite

UnifyFS, a user-level file system specifically designed for burst buffers, aims to integrate the existing node-local storage layers in parallel computing systems, thereby providing a unified namespace for parallel computing applications. This study details the successful integration of UnifyFS with the Slurm resource manager, and emphasizes the synergistic effects of this integration in enhancing the performance of parallel computing systems. Detailed evaluations of this integration on leading parallel computing systems show that the integrated system exhibits significant advantages in scalability of data writing operations, and achieves notable performance improvements in the read-write bandwidth of applications, bringing higher efficiency and improvements to parallel computing applications. The paper thoroughly analyzes the effective integration mechanisms of UnifyFS and Slurm, and explores its potential in optimizing the collaborative operation of distributed storage and task scheduling, offering a viable solution to the broad parallel computing community. Future research will focus on further optimizing this integration framework, adding more advanced features, to achieve broader applications and technological innovations.

show abstract

Energy-Aware Checkpointing Strategies

Aupy

Benoît

Diouri

et al. 2015

Computer Communications and Networks

View full text Add to dashboard Cite

HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

A 1 PB/s file system to checkpoint three million MPI tasks

Cited by 16 publications

References 19 publications

ONFS: a hierarchical hybrid file system based on memory, SSD, and HDD for high performance computers

ONFS: a hierarchical hybrid file system based on memory, SSD, and HDD for high performance computers

Research on the integration of UnifyFS and Slurm

Energy-Aware Checkpointing Strategies

Contact Info

Product

Resources

About