A Study of Job Failure Prediction on Supercomputers with Application Semantic Enhancement

Zhang, Haotong; Xian, Gang; Yang, Wei; Yu, Jie

doi:10.5626/jcse.2022.16.4.222

Cited by 2 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the studies by Banjongkan et al 19 and Yoo et al, 20 although they employ tree structure models, their algorithms are single, and they do not consider the correlation of job application sequences as addressed in this article, leading to weaker predictive performance. Furthermore, compared to our previous research work, 21,22 the FP-JSC framework has achieved a promising prediction effect.…”

Section: Comparison With Other Methodsmentioning

confidence: 74%

“…In this article, we are attempting to construct a machine learning model with rapid computing efficiency, robust interpretability, and no prior knowledge assumptions. In our previous research, 21,22 it was demonstrated that tree structure models' learning algorithms are better suited for our HPC system. Hence, we have opted for the following three learning algorithms.…”

Section: Tree Structure Algorithmmentioning

confidence: 98%

“…For instance, the job log records a large number of similar job names. 22 As illustrated in Table 4, these jobs may originate from the same user's research application, and they may exhibit similar working patterns, which can provide some insight into the likelihood of job failures. To protect user privacy, we cannot directly access job applications.…”

Section: Job Applicationmentioning

confidence: 99%

“…Each user submits multiple jobs based on their research direction on the production supercomputer. For instance, the job log records a large number of similar job names 22 . As illustrated in Table 4, these jobs may originate from the same user's research application, and they may exhibit similar working patterns, which can provide some insight into the likelihood of job failures.…”

Section: Application Sequence and Framework Designmentioning

confidence: 99%

See 3 more Smart Citations

FP‐JSC: Job failure prediction on supercomputers through job application sequence correlation

Xian,

Yang,

2024

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SummarySupercomputers are advanced computing systems interconnected through high‐speed communication networks, consisting of independent computational nodes. During the unfolding of the big data era, the potent computational capabilities of these supercomputers play a pivotal role in scientific computing. Despite executing numerous advanced computational science and engineering tasks on supercomputers, many submitted jobs fail due to various factors, resulting in user inefficiencies. These failures not only consume system resources but also reduce the overall efficiency of the system. Previous research often couples job performance features with a single machine learning method for predicting job failure. However, a primary hurdle emerges from the high cost of gathering these features, complicating their real‐world applicability. To address this challenge, our study establishes correlations among job applications through extensive job log analysis. Leveraging correlations, we propose a predictive framework based on job application sequence correlation (called FP‐JSC). This innovative framework employs multiple machine learning models to offer holistic predictions, selecting the most suitable model based on its learning effectiveness. Moreover, the framework optimizes feature collection expenses without adversely affecting job execution. We determine job applications using both job paths and job names, with the former emerging as a novel feature derived from supplementary monitoring data. Empirical results underscore FP‐JSC's effectiveness, accurately identifying over 89% of jobs with 95% specificity and 89% sensitivity—outperforming single prediction methods employed in related works.

show abstract