Feedback-Based Resource Allocation for Batch Scheduling of Scientific Workflows

Witt, Carl; Wagner, Dennis; Leser, Ulf

doi:10.1109/hpcs48598.2019.9188055

Cited by 15 publications

(18 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, estimation methods for the usage of other resources, e.g. main memory or network bandwidth, often apply similar techniques [43,45].…”

Section: Related Workmentioning

confidence: 99%

“…Both approaches have in common that they need at least comprehensive knowledge about execution times of all tasks on all available nodes. However, these values are not available in advance but must be determined either by asking users for estimates [18,22,23], by analyzing historical traces [35,36,42], or by using some form of online learning [43,45]. Lotaru aims to estimate the runtime for all task-node pairs in a cluster to enable the use of existing scheduling methods in real-world systems.…”

Section: Scheduling Workflow Tasks Onto Heterogeneous Clustersmentioning

confidence: 99%

“…Scientists from many domains, such as bioinformatics, remote sensing, and physics, use scientific workflow management systems to define, compose, and reproducibly execute their data analysis pipelines over large datasets [13,14,45]. These workflows are commonly organized as a directed acyclic graph (DAG), consisting of a set of abstract tasks T and a set of directed edges E. While the abstract tasks serve as templates for their physical instances on real datasets, edges describe the flow of data between tasks and thus constrain the order of execution and degree of parallelism of task executions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

Bader,

Lehmann,

Thamsen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Many scientific workflow scheduling algorithms need to be informed about task runtimes a-priori to conduct efficient scheduling. In heterogeneous cluster infrastructures, this problem becomes aggravated because these runtimes are required for each task-node pair. Using historical data is often not feasible as logs are typically not retained indefinitely and workloads as well as infrastructure changes. In contrast, online methods, which predict task runtimes on specific nodes while the workflow is running, have to cope with the lack of example runs, especially during the start-up.In this paper, we present Lotaru, a novel online method for locally estimating task runtimes in scientific workflows on heterogeneous clusters. Lotaru first profiles all nodes of a cluster with a set of shortrunning and uniform microbenchmarks. Next, it runs the workflow to be scheduled on the user's local machine with drastically reduced data to determine important task characteristics. Based on these measurements, Lotaru learns a Bayesian linear regression model to predict a task's runtime given the input size and finally adjusts the predicted runtime specifically for each task-node pair in the cluster based on the micro-benchmark results. Due to its Bayesian approach, Lotaru can also compute robust uncertainty estimates and provides them as an input for advanced scheduling methods.Our evaluation with five real-world scientific workflows and different datasets shows that Lotaru significantly outperforms the baselines in terms of prediction errors for homogeneous and heterogeneous clusters. CCS CONCEPTS• Information systems → Information systems applications; • Computer systems organization → Distributed architectures; • Software and its engineering → Software architectures.

show abstract

“…However, estimation methods for the usage of other resources, e.g. main memory or network bandwidth, often apply similar techniques [43,45].…”

Section: Related Workmentioning

confidence: 99%

Section: Scheduling Workflow Tasks Onto Heterogeneous Clustersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

Bader,

Lehmann,

Thamsen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…With the help of resource managers like Kubernetes [7], Slurm [8], HTCondor [9] or Yarn [10] tasks are scheduled to the infrastructure components. This is necessary since workflows can consist of a large number of tasks, which are often recurring [6], [11], [12]. This specific pattern, combined with large amounts of data, leads to long runtimes on clusters such as several days or weeks [11], [12].…”

Section: Introduction Scientific Workflow Management Systems (Swms) Likementioning

confidence: 99%

“…This is necessary since workflows can consist of a large number of tasks, which are often recurring [6], [11], [12]. This specific pattern, combined with large amounts of data, leads to long runtimes on clusters such as several days or weeks [11], [12]. Therefore, data-parallel computing on large scale-outs is needed to increase the throughput and to ensure that the analysis executes in a certain timeframe.…”

Section: Introduction Scientific Workflow Management Systems (Swms) Likementioning

confidence: 99%

Tarema: Adaptive Resource Allocation for Scalable Scientific Workflows in Heterogeneous Clusters

Bader

Thamsen

Kulagina

et al. 2021

2021 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Scientific workflow management systems like Nextflow support large-scale data analysis by abstracting away the details of scientific workflows. In these systems, workflows consist of several abstract tasks, of which instances are run in parallel and transform input partitions into output partitions. Resource managers like Kubernetes execute such workflow tasks on cluster infrastructures. However, these resource managers only consider the number of CPUs and the amount of available memory when assigning tasks to resources; they do not consider hardware differences beyond these numbers, while computational speed and memory access rates can differ significantly.We propose Tarema, a system for allocating task instances to heterogeneous cluster resources during the execution of scalable scientific workflows. First, Tarema profiles the available infrastructure with a set of benchmark programs and groups cluster nodes with similar performance. Second, Tarema uses online monitoring data of tasks, assigning labels to tasks depending on their resource usage. Third, Tarema uses the node groups and task labels to dynamically assign task instances evenly to resources based on resource demand. Our evaluation of a prototype implementation for Kubernetes, using five real-world Nextflow workflows from the popular nf-core framework and two 15-node clusters consisting of different virtual machines, shows a mean reduction of isolated job runtimes by 19.8% compared to popular schedulers in widely-used resource managers and 4.54% compared to the heuristic SJFN, while providing a better cluster usage. Moreover, executing two long-running workflows in parallel and on restricted resources shows that Tarema is able to reduce the runtimes even more while providing a fair cluster usage.

show abstract

Leveraging Reinforcement Learning for Task Resource Allocation in Scientific Workflows

Bader

Zunker

Becker

et al. 2022

2022 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Feedback-Based Resource Allocation for Batch Scheduling of Scientific Workflows

Cited by 15 publications

References 25 publications

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

Tarema: Adaptive Resource Allocation for Scalable Scientific Workflows in Heterogeneous Clusters

Leveraging Reinforcement Learning for Task Resource Allocation in Scientific Workflows

Contact Info

Product

Resources

About