Adapting Batch Scheduling to Workload Characteristics: What Can We Expect From Online Learning?

Abstract

Despite the impressive growth and size of super-computers, the computational power they provide still cannot match the demand. Efficient and fair resource allocation is a critical task. Super-computers use Resource and Job Management Systems to schedule applications, which is generally done by relying on generic index policies such as First Come First Served and Shortest Processing time First in combination with Backfilling strategies. Unfortunately, such generic policies often fail to exploit specific characteristics of real workloads. In this work, we focus on improving the performance of online schedulers. We study mixed policies, which are created by combining multiple job characteristics in a weighted linear expression, as opposed to classical pure policies which use only a single characteristic. This larger class of scheduling policies aims at providing more flexibility and adaptability. We use space coverage and black-box optimization techniques to explore this new space of mixed policies and we study how can they adapt to the changes in the workload. We perform an extensive experimental campaign through which we show that (1) even the best pure policy is far from optimal and that (2) using a carefully tuned mixed policy would allow to significantly improve the performance of the system. (3) We also provide empirical evidence that there is no one size fits all policy, by showing that the rapid workload evolution seems to prevent classical online learning algorithms from being effective.

Improving the performance of batch schedulers using online job runtime classification

“…WFP has cumulative values close to SPF and SAF, while FCFS has the worst values by a large margin in all cases. These results are consistent with previous comparisons of scheduling policies [14], [15].…”

“…Finally, they argue that training can take many months (or years) before it reaches a stable level when using a few features, which would prevent practical deployments. We also observed strong day-to-day performance variability [15] and the potential inefficiency of static policies learned from long past periods of time. These observations motivate the need for reactive online learning policy that can quickly adapt to rapid load variations.…”

“…Starvation occurs when a job waits for an indefinite or a very long time in the queue. However, some studies [13], [15] show that SAF and SPF provide better results on performance metrics in almost all cases. Furthermore, one can prevent starvation by putting a threshold on the waiting time [14].…”

“…Yet, they only focus on manipulating the backfilling policy (replacing FCFS with SPF) and do not explore the effects of changing the main index policy (FCFS). Later works [14], [15] show that the main ordering policy has a more significant impact on the general scheduling performance. A problem all the aforementioned prediction-based approaches frequently suffer from is the underestimation of running times.…”

“…Interestingly, the generated functions resemble the Smallest Area First (SAF) policy. Legrand et al [15] showed that using a linear combination of job characteristics allows building index policies that can significantly improve system performance. The authors also showed that the continuously changing nature of the data makes it very hard to learn online the optimal weights for this linear combination, preventing any static policy to be fully effective.…”