The Design Space of Emergent Scheduling for Distributed Execution Frameworks

Abstract

Distributed Execution Frameworks (DEFs) such as Apache Spark have become ubiquitous as a solution for the execution of user-defined jobs to process terabytes of data across hundreds of nodes. One of the key costs of DEFs is scheduling of which parts of each job are placed on each host; better scheduling decisions provide lower overall execution time for each job, more efficient resource usage, and reduced energy consumption. Existing DEFs use a static approach to scheduling, either with a single generalised scheduler which aims to be a good fit for most workloads, or with a special-purpose scheduler which is tuned to optimise for a particular kind of workload. In both cases the scheduling implementation is fixed at design-time such that the DEF is unable to adjust to the actual characteristics of workloads that arrive at deployment time. In this paper we introduce an emergent scheduler for Distributed Execution Frameworks. This scheduler can be composed and re-composed at runtime from a set of different building blocks, allowing the system to dynamically provide the benefits of differing scheduling policies over time depending on the actual properties of incoming workloads -with improved performance and resource usage. In this paper we present the overall design of our emergent scheduler, we discuss the theoretical design space of different scheduling approaches, and we examine a specific research question to determine the correlation between workload properties and scheduling performance for different scheduler implementations. Our results are based on a real implementation of our emergent DEF running across multiple hosts in a real datacentre, and our implementation is made available as open-source software.

Hatch: Self-distributing systems for data centers

“…The use of Hatch in batch processing systems introduces potential challenges in the measurement of ongoing performance in order to make rapid progress in finding an ideal solution; in some cases incremental progress through a batch job can serve this purpose (e.g. [10], but identifying effective sub-batch measurement approaches can be challenging. On the specific optimization metric used in this work, to date we have only experimented with response time as a metric and have only examined a single metric.…”

Code and Data Synthesis for Genetic Improvement in Emergent Software Systems

“…Emergent software systems aim to bring these ideas to everyday software, such as web servers, caches and databases, big data processing frameworks, and microservices [Dean and Porter 2021;Filho et al 2018;Rodrigues-Filho and Porter 2022]. Emergent software combines the general ability to wire units of logic together to form a system, and rewire those units safely while a system is running, with real-time machine learning which quantifies the current operating environment and learns online which combination of building blocks has the highest utility in each environment.…”