PREP: Predicting Job Runtime with Job Running Path on Supercomputers

Zhou, Longfang; Zhang, Xiaorong; Yang, Wenxiang; Han, Yongguo; Wang, Fang; Wu, Yadong; Yu, Jie

doi:10.1145/3472456.3473521

Cited by 8 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are many types of job attributes collected by the supercomputer center, which cannot all be used by the algorithm. Therefore, according to the job attribute standard provided by the research in literature 11,12 , 10 attributes in the job are selected.…”

Section: Job Information Selectionmentioning

confidence: 99%

Job running time prediction algorithm based on classification and ensemble learning

Kong

Zhang

Zheng

et al. 2023

Second International Conference on Algorithms, Microchips, and Network Applications (AMNA 2023)

View full text Add to dashboard Cite

Backfill scheduling is a common scheduling strategy in high-performance computing systems that allows priority execution of low-priority jobs to make better use of available resources. Job running time is an important parameter that affects the performance of backfill scheduling algorithm. However, in order to avoid job killing due to lack of time, the running time requested by users is often several times higher than the actual running time, resulting in a certain degree of resource waste. In order to improve resource utilization, a new job running time prediction algorithm is proposed by combining classification and ensemble learning methods. The algorithm first classifies the historical job set according to the application type, then uses Jaccard coefficient to calculate the similarity between the jobs, and further classifies the jobs. At the same time, different integration models are constructed for the jobs of different application types. New jobs are categorized, and the class's integration model is used to predict the running time of the new job. The algorithm was tested on the historical job data of the National Supercomputing Center Kunshan, Hefei Advanced Computing Center and "Wuzhen Light" supercomputing Center and compared with GA-sim algorithm and IRPA algorithm. The experimental results show that compared with the IRPA algorithm, the average absolute error of the algorithm is improved by 60% on the three data sets on average. Compared with the GA-sim algorithm, the average prediction accuracy of the algorithm is improved by 20% on the three data sets on average. Through the in-depth analysis of the experimental results, the amplification method for the low estimation of long and short jobs is given.

show abstract

Section: Job Information Selectionmentioning

confidence: 99%

Job running time prediction algorithm based on classification and ensemble learning

Kong

Zhang

Zheng

et al. 2023

Second International Conference on Algorithms, Microchips, and Network Applications (AMNA 2023)

View full text Add to dashboard Cite

show abstract

“…In this experiment, we evaluate the performance of the ESLURM's job runtime estimation framework using historical We compare ESLURM with a variety of runtime prediction models. Among them, Last-2 [40], IRPA [51], TRIP [52] and PREP [53] are the latest related work. As shown in Fig.…”

Section: E Performance Of Job Runtime Estimationmentioning

confidence: 99%

“…[52] proposes an online adjustment framework, TRIP, which utilizes the data truncation capability of Tobit regression to obtain accurate runtime estimates. PREP [53] is a runtime prediction framework which groups jobs into several clusters according to their running paths and trains a runtime prediction model for each job cluster.…”

Section: Related Workmentioning

confidence: 99%

Towards Scalable Resource Management for Supercomputers

Dai

Dong

Lü

et al. 2022

SC22: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Today's supercomputers offer massive computation resources to execute a large number of user jobs. Effectively managing such large-scale hardware parallelism and workloads is essential for supercomputers. However, existing HPC resource management (RM) systems fail to capitalize on the hardware parallelism by following a centralized design used decades ago. They give poor scalability and inefficient performance on today's supercomputers, which will worsen in exascale computing. We present ESLURM, a better RM for supercomputers. As a departure from existing HPC RMs, ESLURM implements a distributed communication structure. It employs a new communication tree strategy and uses job runtime estimation to improve communications and job scheduling efficiency. ESlurm is deployed into production in a real supercomputer. We evaluate ESLURM on up to 20K nodes. Compared to state-of-the-art RM solutions, ESLURM exhibits better scalability, significantly reducing the resource usage of master nodes and improving data transfer and job scheduling efficiency by a large margin.

show abstract

PreF: Predicting job failure on supercomputers with job path and user behavior

Xian

Zhang

et al. 2022

Concurrency and Computation

View full text Add to dashboard Cite

Large numbers of jobs are executed on supercomputers almost every day. Unfortunately, many jobs would fail for various reasons, resulting in the waste of resources and the prolonged waiting time for queuing jobs. Job failure prediction can guide adjustment measures in advance, which is vital to the system's overall execution efficiency and reliability. Aiming at the problem that the existing job failure prediction methods are single, the collection of job features is complex and challenging to apply. This article strives to study whether these failed jobs can be predicted with known and synthetic features. We perform a comprehensive analysis of large amounts of historical data and various features and find that two novel features (running path and retry count) can predict job failure well. The running path indicates the application type a job belongs to, and the retry count reflects the user's behavior when the job fails. We propose a job failure prediction framework called PreF on supercomputers using machine learning based on the novel features. The experimental results show that PreF can correctly identify over 89% of jobs, outperforming the latest related methods on the comprehensive evaluation indicator (S_score) by around 4%.

show abstract

PREP: Predicting Job Runtime with Job Running Path on Supercomputers

Cited by 8 publications

References 13 publications

Job running time prediction algorithm based on classification and ensemble learning

Job running time prediction algorithm based on classification and ensemble learning

Towards Scalable Resource Management for Supercomputers

PreF: Predicting job failure on supercomputers with job path and user behavior

Contact Info

Product

Resources

About