Flux: A Next-Generation Resource Management Framework for Large HPC Centers

Ahn, Dong H.; Garlick, Jim; Grondona, Mark; Lipari, D A; Springmeyer, Becky; Schulz, Martin

doi:10.1109/icppw.2014.15

Cited by 50 publications

(23 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fortunately, demand has grown to have an RJMS to schedule jobs at levels above these boundaries using hierarchical approaches. Flux [4] is among a few of the already available next-generation resource and job management software systems. Being developed at Lawrence Livermore National Laboratory (LLNL), Flux responds to the aforementioned need and presents an opportunity for our methodology to be developed.…”

Section: Global Viewmentioning

confidence: 99%

“…In the case of I/O-aware scheduling, a resource model must also capture the essential attributes of the I/O subsystem of clusters in addition to their compute resources (e.g., compute nodes and cores). We use the Resource Description Language (RDL) within Flux [4] to model the I/O subsystems as a hierarchy of network switches and gateway nodes leading to a parallel file system (PFS) and establish hierarchical relationships between them. Figure 2 shows a simple example of an RDL-based model with its compute nodes, each with a BB at the fringes of the hierarchy.…”

Section: I/o Subsystem Modelingmentioning

confidence: 99%

“…A center-wide resource and job management framework enables schedulers to model the global I/O subsystem and view jobs beyond cluster boundaries, enabling global scheduling decisions for shared file systems. We integrate the knowledge of the I/O subsystem including burst bu↵ers and I/O-aware algorithms directly into Flux [4], one of the few resource and job managers for next-generation centers. Exploiting a global view in Flux, however, comes with multidimensional scale challenges.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

Herbein

Ahn

Lipari

et al. 2016

Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

The economics of flash vs. disk storage is driving HPC centers to incorporate faster solid-state burst bu↵ers into the storage hierarchy in exchange for smaller parallel file system (PFS) bandwidth. In systems with an underprovisioned PFS, avoiding I/O contention at the PFS level will become crucial to achieving high computational e ciency. In this paper, we propose novel batch job scheduling techniques that reduce such contention by integrating I/O awareness into scheduling policies such as EASY backfilling. We model the available bandwidth of links between each level of the storage hierarchy (i.e., burst bu↵ers, I/O network, and PFS), and our I/O-aware schedulers use this model to avoid contention at any level in the hierarchy. We integrate our approach into Flux, a next-generation resource and job management framework, and evaluate the e↵ectiveness and computational costs of our I/O-aware scheduling. Our results show that by reducing I/O contention for underprovisioned PFSes, our solution reduces job performance variability by up to 33% and decreases I/O-related utilization losses by up to 21%, which ultimately increases the amount of science performed by scientific workloads.

show abstract

Section: Global Viewmentioning

confidence: 99%

Section: I/o Subsystem Modelingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

Herbein

Ahn

Lipari

et al. 2016

Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…The problem addressed in this paper is the one faced by Resource and Job Management Systems (RJMS) such as SLURM [2], PBS [1] and OAR [7] and more recently by Flux [4].…”

Section: System Descriptionmentioning

confidence: 99%

Tuning EASY-Backfilling Queues

Lelong

Reis

Trystram

2018

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

Abstract. EASY-Backfilling is a popular scheduling heuristic for allocating jobs in large scale High Performance Computing platforms. While its aggressive reservation mechanism is fast and prevents job starvation, it does not try to optimize any scheduling objective per se. We consider in this work the problem of tuning EASY using queue reordering policies. More precisely, we propose to tune the reordering using a simulationbased methodology. For a given system, we choose the policy in order to minimize the average waiting time. This methodology departs from the First-Come, First-Serve rule and introduces a risk on the maximum values of the waiting time, which we control using a queue thresholding mechanism. This new approach is evaluated through a comprehensive experimental campaign on five production logs. In particular, we show that the behavior of the systems under study is stable enough to learn a heuristic that generalizes in a train/test fashion. Indeed, the average waiting time can be reduced consistently (between 11% to 42% for the logs used) compared to EASY, with almost no increase in maximum waiting times. This work departs from previous learning-based approaches and shows that scheduling heuristics for HPC can be learned directly in a policy space.

show abstract

“…A second example is Moab [19] scheduler, that implements a simulator mode, in which the user can interact and control simulated time, but it is a proprietary software. Flux [20] is a Resource Management framework that includes a simulator in its code, but either publications and documentation lack of information about it. There is no published evaluation of the consistency and the accuracy for any of these simulators.…”

Section: Related Workmentioning

confidence: 99%

Evaluating SLURM Simulator with Real-Machine SLURM and Vice Versa

Jokanović

D'Amico

Corbalan

2018

2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)

View full text Add to dashboard Cite

Having a precise and a fast job scheduler model that resembles the real-machine job scheduling software behavior is extremely important in the field of job scheduling. The idea behind SLURM simulator is preserving the original code of the core SLURM functions while allowing for all the advantages of a simulator. Since 2011, SLURM simulator has passed through several iterations of improvements in different research centers. In this work, we present our latest improvements of SLURM simulator and perform the first-ever validation of the simulator on the real machine. In particular, we improved the simulator's performance for about 2.6 times, made the simulator deterministic across several same setup runs, and improved the simulator's accuracy; its deviation from the real-machine is lowered from previous 12% to at most 1.7%. Finally, we illustrate with several use cases the value of the simulator for job scheduling researchers, SLURM-system administrators, and SLURM developers.

show abstract

Flux: A Next-Generation Resource Management Framework for Large HPC Centers

Cited by 50 publications

References 16 publications

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

Tuning EASY-Backfilling Queues

Evaluating SLURM Simulator with Real-Machine SLURM and Vice Versa

Contact Info

Product

Resources

About