Batch queue resource scheduling for workflow applications

Abstract

Workflow computations have become a major programming paradigm for scientific applications. However, acquiring enough computational resources to execute a workflow poses a challenge in a batch queue controlled resource due to the space-sharing nature of the resource management policy. This paper introduces a scheduling technique that aggregates a workflow application into several subcomponents. It then uses the batch queue to acquire resources for each subcomponent, overlapping resource provisioning overhead (wait time) of one with the execution of others. We implemented a prototype of this technique and tested it using five high performance computing centers job submission logs. The results show that our approach can eliminate as much as 70% of the wait time over more traditional techniques that request resources for individual workflow nodes or that acquire all the resources for the whole workflow at once.

Level based batch scheduling strategy with idle slot reduction under DAG constraints for computational grid

“…Therefore, many heuristics for scheduling have been proposed to gain sub-optimal solutions to the problem. These heuristics can be either static with scheduling decisions being made prior to execution enabling the scheduler to make efficient scheduling decisions due to prior knowledge of the resources and application (Braun et al, 2001;Daoud and Kharma, 2008;Maheswaran et al, 1999b;Papazachos and Karatza, 2010), dynamic where scheduling decisions are made during the execution making it more challenging due to the dynamism involved in the resources as well as jobs (Maheswaran and Siegel, 1998;Maheswaran et al, 1999a;Wang et al, 2005), immediate mode in which the jobs are dispatched as soon as they are receive (Ruyan et al, 2009;Xhafa et al, 2007b) or the batch mode where first a group of jobs is formed followed by scheduling leading to collective scheduling decisions being made for better mapping of the jobs (Cai et al, 2002;Xhafa et al, 2007a;Leah et al, 2006;Ran and Han, 2009;Zhang et al, 2009). …”

“…Workflow computations are a major programming paradigm for scientific applications. A scheduling scheme has been proposed in Zhang et al (2009) that aggregates a workflow scheme and a batch queue to acquire resources for each sub component with an approach to reduce the wait time. Most of the work reported schedules the batch while considering only a few parameters leaving scope for the problem to be reconsidered using other appropriate parameters (Braun et al, 2001;Cai et al, 2002;Ran and Han, 2009;Ruyan et al, 2009) while in some other places in the literature mapping done to optimize even more than one parameter has also been reported (Xhafa et al, 2007a,b).…”

A precedence based load balancing strategy for batch of DAGs for computational grid

“…Some of the list scheduling heuristics for DAGs is also reported for generating better schedules like Heterogeneous Earliest Finish Time (HEFT), Critical path on Processor (CPOP) [12] and Levelized Min time (LMT) [13]. Further, some methods in which jobs represented as work flow application are partitioned into groups according to level and then scheduling done level wise with a focus to find optimal schedule at each depth level [13,14,15]. A Polynomial Time Approximation Schemes (PTAS) for bounded batch scheduling with the objective of minimizing the total completion time while considering the precedence constraints is also presented [16].…”

ASA - The Adaptive Scheduling Architecture

“…Thus, for a perstate management to have lower makespans, the accumulated sum of its waiting times 𝑞 ′ 𝑖 has to be lower than the single waiting time 𝑞 1 in the 𝐵𝑖𝑔𝐽𝑜𝑏 strategy, i.e., 𝑠 𝑖=1 𝑞 ′ 𝑖 < 𝑞 1 . One strategy to achieve this is to heuristically pack multiple stages within medium-sized job submissions [31], though it may not achieve optimal resource usage. Finally, as the queue waiting time is a system parameter controlled by the resource manager, another natural strategy for the users is to observe its behaviour and estimate it.…”