Learning-Based Approaches to Estimate Job Wait Time in HTC Datacenters

Gombert, Luc; Suter, Frédéric

doi:10.1007/978-3-030-88224-2_6

Cited by 4 publications

(1 citation statement)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The learning time measures the time it takes to process the whole set of historical data and build its prediction model. It depends on the complexity of the method, the size of the training data as well as the computing resource available for model training [10]. The prediction time is the time it takes the model to infer the wait time for a newly submitted job.…”

Section: A Performance Metricsmentioning

confidence: 99%

Queue wait time prediction in High PerformanceComputing (HPC) Systems

Okafor,

Lusch,

Vishwanath

2024

Preprint

View full text Add to dashboard Cite

High Performance Computing (HPC) systems are critical enablers for groundbreaking scientific research across various domains. Efficient resource allocation, facilitated by job scheduling, is paramount for maximizing the utilization of HPC systems. However, the variability in wait times for queued jobs poses challenges for users, necessitating accurate job wait time estimation. This paper explores the influence of job characteristics, including job size (the number of nodes requested and walltime), the queue to which the job is submitted and other resource requirements, on job wait times in leadership-class HPC systems. Focusing on the Theta Cray XC40 and Polaris machines at Argonne National Laboratory, the study evaluates the performance of different supervised learning algorithms in predicting job wait times. It underscores the importance of effective data handling and processing, including outlier detection and Principal Component Analysis (PCA), to enhance prediction accuracy.The findings reveal insights into the relationship between job characteristics and wait times, offering a foundation for optimizing resource allocation and enhancing user experience. The methodologies and tools developed in this study are adaptable to other leadership-class HPC systems, providing a valuable contribution to the broader HPC community aiming to improve job scheduling efficiency and user satisfaction.

show abstract

Section: A Performance Metricsmentioning

confidence: 99%