Flux: Overcoming scheduling challenges for exascale workflows

Ahn, Dong H.; Bass, Ned; Chu, A.; Garlick, Jim; Grondona, Mark; Herbein, Stephen; Ingólfsson, Helgi I.; Koning, J. M.; Patki, Tapasya; Scogland, Thomas R. W.; Springmeyer, Becky; Taufer, Michela

doi:10.1016/j.future.2020.04.006

Cited by 46 publications

(20 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We rely on the Flux production workload manager given by Herbein et al (2016), Ahn et al (2020) to study a HPC environment augmented with AI4IO. To this end, we extend the original Flux workload manager and its simulator to incorporate PRIONN and CanarIO.…”

Section: Io-aware Flux-based Testing Environmentmentioning

confidence: 99%

AI4IO: A suite of AI-based tools for IO-aware scheduling

Wyatt

Herbein

Gamblin

et al. 2022

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

Traditional workload managers do not have the capacity to consider how IO contention can increase job runtime and even cause entire resource allocations to be wasted. Whether from bursts of IO demand or parallel file systems (PFS) performance degradation, IO contention must be identified and addressed to ensure maximum performance. In this paper, we present AI4IO (AI for IO), a suite of tools using AI methods to prevent and mitigate performance losses due to IO contention. AI4IO enables existing workload managers to become IO-aware. Currently, AI4IO consists of two tools: PRIONN and CanarIO. PRIONN predicts IO contention and empowers schedulers to prevent it. CanarIO mitigates the impact of IO contention when it does occur. We measure the effectiveness of AI4IO when integrated into Flux, a next-generation scheduler, for both small- and large-scale IO-intensive job workloads. Our results show that integrating AI4IO into Flux improves the workload makespan up to 6.4%, which can account for more than 18,000 node-h of saved resources per week on a production cluster in our large-scale workload.

show abstract

Section: Io-aware Flux-based Testing Environmentmentioning

confidence: 99%

AI4IO: A suite of AI-based tools for IO-aware scheduling

Wyatt

Herbein

Gamblin

et al. 2022

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

show abstract

“…This parent–child relationship can extend to an arbitrary depth and width, creating many opportunities for scheduling parallelization. This alternative model has already proven to be effective on emerging HPC workflows Ahn et al (2020); Di Natale et al (2019); Peterson et al (2019). Unfortunately, it lacks a formal definition.…”

Section: Generalized Hierarchical Schedulingmentioning

confidence: 99%

“…Generalized Hierarchical Scheduling (GHS) addresses these scalability and customization challenges systematically, generically, and portably Ahn et al (2020); Di Natale et al (2019); Peterson et al (2019). Figure 1 compares GHS to the typical HPC centralized scheduling paradigm.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we address this performance modeling challenge. Our contributions are:• A formal definition of GHS and its underlying principles;• An analytical performance model to capture the performance properties of GHS;• Validation of our analytical model on Flux Ahn et al (2020), a production scheduling software that embodies GHS, with four different ensemble applications;• Demonstration of our validated analytic model as a tool for GHS performance exploration on extreme-scale systems; and• Initial insights into efficiently selecting the optimal GHS tree for applications running on pre-exascale systems…”

Section: Introductionmentioning

confidence: 99%

“…• Validation of our analytical model on Flux Ahn et al (2020), a production scheduling software that embodies GHS, with four different ensemble applications;…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An analytical performance model of generalized hierarchical scheduling

Herbein

Patki

Ahn

et al. 2022

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

High performance computing (HPC) workflows are undergoing tumultuous changes, including an explosion in size and complexity. Despite these changes, most batch job systems still use slow, centralized schedulers. Generalized hierarchical scheduling (GHS) solves many of the challenges that face modern workflows, but GHS has not been widely adopted in HPC. A major difficulty that hinders adoption is the lack of a performance model to aid in configuring GHS for optimal performance on a given application. We propose an analytical performance model of GHS, and we validate our proposed model with four different applications on a moderately-sized system. Our validation shows that our model is extremely accurate at predicting the performance of GHS, explaining 98.7% of the variance (i.e., an R2 statistic of 0.987). Our results also support the claim that GHS overcomes scheduling throughput problems; we measured throughput improvements of up to 270× on our moderately-sized system. We then apply our performance model to a pre-exascale system, where our model predicts throughput improvements of four orders of magnitude and provides insight into optimally configuring GHS on next generation systems.

show abstract