Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation

Abstract

Monitoring and analyzing the execution of a workload is at the core of the operation of data centers. It allows operators to verify that the operational objectives are satised or detect and react to any unexpected and unwanted behavior. However, the scale and complexity of large workloads composed of millions of jobs executed each month on several thousands of cores, often limit the depth of such an analysis. This may lead to overlook some phenomena that, while not harmful at a global scale, can be detrimental to a specic class of users. In this paper, we illustrate such a situation by analyzing a large High Throughput Computing (HTC) workload trace coming from one of the largest academic computing centers in France. The Fair-Share algorithm at the core of the batch scheduler ensures that all user groups are fairly provided with an amount of computing resources commensurate to their expressed needs. However, a deeper analysis of the produced schedule, especially of the job waiting times, shows a certain degree of unfairness between user groups. We identify the conguration of the quotas and scheduling queues as the main root causes of this unfairness. We thus propose a drastic reconguration of the system that aims at being more suited to the characteristics of the workload and at better balancing the waiting time among user groups. We evaluate the impact of this reconguration through detailed simulations. The obtained results show that it still satises the main operational objectives while signicantly improving the quality of service experienced by formerly unfavored users.

Alea – Complex Job Scheduling Simulator

“…Also users and/or groups are often subject to a upper bound on the amount of resources they can use simultaneously. For this purpose, Alea provides CPU quotas, that guarantee that a user/group will not exceed the corresponding maximum allowed share of resources [2].…”

“…Using Alea, we were able to model the system and evaluate new setups for the system's queues and the per-group CPU quotas. This new setup allowed for improved fairness for local users, by better balancing their wait times with the wait times of grid-originating jobs [2].…”

“…Since then, many new features have been implemented and the simulator has been successfully used for various purposes, both as a purely research tool as well as when testing new setups and new scheduling policies for production HPC and HTC systems. The main contribution of this paper is that (1) we describe recent improvements in the simulator, that allow for truly complex simulations that involve several detailed setups that correspond to typical real-life based scenarios, (2) we describe the recent speedup of the simulator that enables us to run truly large-scale simulations involving millions of jobs and thousands of nodes that complete in just a few hours, and (3) we provide several real-life based case studies where Alea has been used to develop and evaluate effects of major modifications of real HPC and HTC systems.…”

Learning-Based Approaches to Estimate Job Wait Time in HTC Datacenters

“…In a previous study we showed that two distinct sub-workloads are executed at CC-IN2P3 [2]. Some jobs are submitted by a small number of large user groups through a Grid middleware, at a nearly constant rate and with an important upstream control of the submissions while Local users from about 60 dierent groups directly submit their jobs to the batch system.…”

“…This workload is composed of 7,749,500 Grid jobs and 5,748,922 Local jobs, for a total of 13,498,422 jobs. Hereafter we focus only on the Local jobs, as they experience larger wait times than Grid jobs [2].…”