Scalable system scheduling for HPC and big data

Abstract

In the rapidly expanding field of parallel processing, job schedulers are the "operating systems" of modern big data architectures and supercomputing systems. Job schedulers allocate computing resources and control the execution of processes on those resources. Historically, job schedulers were the domain of supercomputers, and job schedulers were designed to run massive, long-running computations over days and weeks. More recently, big data workloads have created a need for a new class of computations consisting of many short computations taking seconds or minutes that process enormous quantities of data. For both supercomputers and big data systems, the efficiency of the job scheduler represents a fundamental limit on the efficiency of the system. Detailed measurement and modeling of the performance of schedulers are critical for maximizing the performance of a large-scale computing system. This paper presents a detailed feature analysis of 15 supercomputing and big data schedulers. For big data workloads, the scheduler latency is the most important performance characteristic of the scheduler. A theoretical model of the latency of these schedulers is developed and used to design experiments targeted at measuring scheduler latency. Detailed benchmarking of four of the most popular schedulers (Slurm, Son of Grid Engine, Mesos, and Hadoop YARN) are conducted. The theoretical model is compared with data and demonstrates that scheduler performance can be characterized by two key parameters: the marginal latency of the scheduler t s and a nonlinear exponent α s . For all four schedulers, the utilization of the computing system decreases to <10% for computations lasting only a few seconds. Multi-level schedulers (such as LLMapReduce) that transparently aggregate short computations can improve utilization for these short computations to >90% for all four of the schedulers that were tested.

A Proposed Architecture for Parallel HPC-based Resource Management System for Big Data Applications

“…It manages different compute jobs related to different users on homogenous or heterogeneous computational resources. It can have different names that reflect the same mechanism such as scheduler, resource manager, resource management system (RMS), and a distributed resource management system (D-RMS) [7]. Despite significant growth in terms of heterogeneity of resources and job complexity and diversity, job schedulers still have the main core function of job queuing, scheduling and resource allocation, and resource management [8] [9].…”

Interactive Launch of 16,000 Microsoft Windows Instances on a Supercomputer

“…Recent experiments have shown that the naive, serial job submission performance of a modern job scheduler can significantly slow down processing for jobs with a very large number of tasks [24]. To achieve maximal job launch performance for large HPC (high performance computing) or HPDA (high performance data analysis) jobs requires employing a technique known as multilevel scheduling which involves modifying our analysis code slightly to be able to process multiple datasets or files with a single job launch [25].…”

“…By leveraging supercomputing and big data storage assets the LLSC has built the MIT SuperCloud, a coherent fusion Fig. 3: Schematic depicting key components of a canonical cluster scheduler including job lifecycle management, resource management, task scheduling and job execution (adapted from [25]). The SLURM scheduler used on the MIT SuperCloud systems behaves according to this model.…”

Measure of Adequacy for the Supercomputer Job Management System Model

“…The passport consists of a parallel program, input data and system requirements (number of cores or nodes, amount of RAM) and execution time limit. S Special software [1] like SLURM [2], PBS [3] or the Russian native job management system SUPPZ [4] manage jobs in supercomputers. The kernel of any job management system (JMS) is the scheduler.…”