A Tool for Prioritizing DAGMan Jobs and its Evaluation

Abstract

Abstract. It is often difficult to perform efficiently a collection of jobs with complex job dependencies due to temporal unpredictability of the grid. One way to mitigate the unpredictability is to schedule job execution in a manner that constantly maximizes the number of jobs that can be sent to workers. A recently developed scheduling theory provides a basis to meet that optimization goal. Intuitively, when the number of such jobs is always large, high parallelism can be maintained, even if the number of workers changes over time in an unpredictable manner. In this paper we present the design, implementation, and evaluation of a practical scheduling tool inspired by the theory. Given a DAGMan input file with interdependent jobs, the tool prioritizes the jobs. The resulting schedule significantly outperforms currently least 13% faster with 95% confidence. An implementation of the tool was integrated with the Condor high-throughput computing system.

Evaluation of Eligible Jobs Maximization Algorithm for DAG Scheduling in Grids

“…Currently available evaluations of PRIO [7] are limited to comparison with FIFO-based ordering. While the latter method, in practice, means 'no prioritization', we consider such an evaluation insufficient.…”

“…An algorithm was introduced to provide Internet-Computing (IC) optimal schedule for a large class of DAGs. However, in practice an algorithm that may be applied for all possible DAGs is needed, so a heuristic algorithm was designed to provide an IC optimal schedule if it exists, while for the rest of DAGs, the heuristics take steps to enhance eligible jobs ratio [7].…”

“…This implementation was used in [7] for comparison with the FIFO ordering. The results of the evaluation were promising, however, while FIFO ordering is rather a basic strategy, we believe that a comparison with stronger algorithms is still needed to understand usefulness of the Malewicz's heuristics.…”

A Monte-Carlo Approach for Full-Ahead Stochastic DAG Scheduling

“…Based on this idea, some heuristics [7], [8] have been developed, which consider only the DAG structure in order to improve application performance. Nevertheless, the evaluation in [9] indicates that, without considering task execution costs when scheduling tasks, the practical effectiveness of these heuristics, at their current form, is limited.…”

On clustering DAGs for task-hungry computing platforms

“…(These speeds can change dynamically, especially when workers are not dedicated to working on the current .) This motivating intuition has been tested in two simulation studies [14], which studies four large real scientific computations, and [11], which studies hundreds of artificially generated computations of varied sizes. Both studies superimpose a computational model on the purely -oriented framework of IC-scheduling (so that one can talk about makespan), and they compare the makespans of simulated executions of s: each tested is executed by IC-scheduling and by a variety of common scheduling heuristics, including the FIFO strategy used by the Condor system 1 .…”