Partitioned Parallel Job Scheduling for Extreme Scale Computing

Brelsford, David; Chochia, George; Falk, Nathan; Marthi, Kailash; Sure, Ravindra; Bobroff, Norman; Fong, Liana; Seelam, Seetharami

doi:10.1007/978-3-642-35867-8_9

Cited by 6 publications

(2 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhou et al 31 modified Slurm to increase scalability and to improve job throughput. Similarly, Brelsford et al 32 developed a scheduling algorithm to improve job scheduling and job dispatching times. In another paper, El‐Ghazawi et al 33 compared throughput and response time of different size jobs using four schedulers used in HPC systems.…”

Section: Related Workmentioning

confidence: 99%

Twister2 Cross‐platform resource scheduler for big data

Uyar

Gunduz

Kamburugamuve

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

Twister2 is an open-source big data hosting environment designed to process both batch and streaming data at scale. Twister2 runs jobs in both high-performance computing (HPC) and big data clusters. It provides a cross-platform resource scheduler to run jobs in diverse environments. Twister2 is designed with a layered architecture to support various clusters and big data problems. In this paper, we present the cross-platform resource scheduler of Twister2. We identify required services and explain implementation details. We present job startup delays for single jobs and multiple concurrent jobs in Kubernetes and OpenMPI clusters. We compare job startup delays for Twister2 and Spark at a Kubernetes cluster. In addition, we compare the performance of terasort algorithm on Kubernetes and bare metal clusters at AWS cloud.

show abstract

Section: Related Workmentioning

confidence: 99%

Twister2 Cross‐platform resource scheduler for big data

Uyar

Gunduz

Kamburugamuve

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Two papers compare job launch time of a monolithic (single) scheduler to that of distributed/partitioned job scheduling. Brelsford et al (2013) explores partitioned parallel job scheduling by modifying the IBM LoadLeveler (a modification of HTCondor) scheduler, while Zhou et al (2013) explores distributed resource-allocation techniques by modifying Slurm. Rather than measuring utilization, both papers measure the throughput of how many jobs they can launch through the scheduler per second.…”

Section: Related Workmentioning

confidence: 99%

Scalable system scheduling for HPC and big data

Reuther

Byun

Arcand

et al. 2018

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

In the rapidly expanding field of parallel processing, job schedulers are the "operating systems" of modern big data architectures and supercomputing systems. Job schedulers allocate computing resources and control the execution of processes on those resources. Historically, job schedulers were the domain of supercomputers, and job schedulers were designed to run massive, long-running computations over days and weeks. More recently, big data workloads have created a need for a new class of computations consisting of many short computations taking seconds or minutes that process enormous quantities of data. For both supercomputers and big data systems, the efficiency of the job scheduler represents a fundamental limit on the efficiency of the system. Detailed measurement and modeling of the performance of schedulers are critical for maximizing the performance of a large-scale computing system. This paper presents a detailed feature analysis of 15 supercomputing and big data schedulers. For big data workloads, the scheduler latency is the most important performance characteristic of the scheduler. A theoretical model of the latency of these schedulers is developed and used to design experiments targeted at measuring scheduler latency. Detailed benchmarking of four of the most popular schedulers (Slurm, Son of Grid Engine, Mesos, and Hadoop YARN) are conducted. The theoretical model is compared with data and demonstrates that scheduler performance can be characterized by two key parameters: the marginal latency of the scheduler t s and a nonlinear exponent α s . For all four schedulers, the utilization of the computing system decreases to <10% for computations lasting only a few seconds. Multi-level schedulers (such as LLMapReduce) that transparently aggregate short computations can improve utilization for these short computations to >90% for all four of the schedulers that were tested.

show abstract

Cooperative and non-cooperative algorithms for distributed parallel jobs scheduling

Behnamian

2022

Flex Serv Manuf J

View full text Add to dashboard Cite

Partitioned Parallel Job Scheduling for Extreme Scale Computing

Cited by 6 publications

References 11 publications

Twister2 Cross‐platform resource scheduler for big data

Twister2 Cross‐platform resource scheduler for big data

Scalable system scheduling for HPC and big data

Cooperative and non-cooperative algorithms for distributed parallel jobs scheduling

Contact Info

Product

Resources

About