NORNS: Extending Slurm to Support Data-Driven Workflows through Asynchronous Data Staging

Abstract

As HPC systems move into the Exascale era, parallel file systems are struggling to keep up with the I/O requirements from data-intensive problems. While the inclusion of burst buffers has helped to alleviate this by improving I/O performance, it has also increased the complexity of the I/O hierarchy by adding additional storage layers each with its own semantics. This forces users to explicitly manage data movement between the different storage layers, which, coupled with the lack of interfaces to communicate data dependencies between jobs in a data-driven workflow, prevents resource schedulers from optimizing these transfers to benefit the cluster's overall performance. This paper proposes several extensions to job schedulers, prototyped using the Slurm scheduling system, to enable users to appropriately express the data dependencies between the different phases in their processing workflows. It also introduces a new service for asynchronous data staging called NORNS that coordinates with the job scheduler to orchestrate data transfers to achieve better resource utilization. Our evaluation shows that a workflow-aware Slurm exploits node-local storage more effectively, reducing the filesystem I/O contention and improving job running times.

Malleability in Modern HPC Systems: Current Experiences, Challenges, and Future Opportunities

“…Moreover, for effective data scheduling, these models could be extrapolated to different node and process configurations. While several services were proposed to support data staging [109]- [112], to the best of our knowledge, none consider these issues.…”

EPCC’s Exascale Journey: A Retrospective of the Past 10 Years and a Vision of the Future

“…Figure 2 shows the project's research around system software support. The research shown here was jointly undertaken by EPCC and Barcelona Supercomputing Centre, and investigated how workflows with data dependencies can be supported on a system with persistent memory [7]. The SLURM resource manager was modified to be aware of the persistent memory, and it was extended to enable users to reconfigure and reboot compute nodes with specific configurations for their jobs.…”

Usage Scenarios for Byte-Addressable Persistent Memory in High-Performance and Data Intensive Computing

“…The advantage of node local storage such as B-APM is that it is possible to leave data on the compute nodes. The NORNS [12] system software layer, which is integrated with the SLURM resource manager, supports the asynchronous staging of data, and the data marshalling between workflow components. As part of the job submission, a user can specify the data dependencies between producer-consumer style workflow components.…”

“…The advantage of node local storage such as B-APM is that it is possible to leave data on the compute nodes. The NORNS [12] App C: Mixed mode with large memory and distributed storage using GekkoFS copy data from fsdax to GekkoFS Fig. 8.…”