Advances in IC-Scheduling Theory: Scheduling Expansive and Reductive Dags and Scheduling Dags via Duality

Abstract

Earlier work has developed the underpinnings of IC-Scheduling Theory, a framework for scheduling computations having intertask dependencies-modeled via dagsfor Internet-based computing. The goal of the schedules produced is to render tasks eligible for execution at the maximum possible rate, with the dual aim of: (a) utilizing remote clients' computational resources well, by always having work to allocate to an available client; (b) lessening the likelihood of a computation's stalling for lack of eligible tasks. The dags handled by the Theory thus far are those that can be decomposed into a given collection of bipartite building-block dags via the operation of dag-decomposition. A basic tool in constructing schedules is a relation ⊲, which allows one to "prioritize" the scheduling of a complex dag's building blocks. The current paper extends IC-Scheduling Theory in two ways: by expanding significantly the repertoire of dags that the Theory can schedule optimally, and by allowing one sometimes to shortcut the algorithmic process required to find optimal schedules. The expanded repertoire now allows the Theory to schedule optimally, among other dags, a large range of dags that are either "expansive," in the sense that they grow outward from their sources, or "reductive," in the sense that they grown inward toward their sinks. The algorithmic shortcuts allow one to "read off" an optimal schedule for a dag from a given optimal schedule for the dag's dual, which is obtained by reversing all arcs (thereby exchanging the roles of sources and sinks).

Evaluation of Eligible Jobs Maximization Algorithm for DAG Scheduling in Grids

“…The heuristic includes the results of research on algorithms, that find optimal IC schedules for some classes of DAGs [3,4,5,6]. PRIO takes advantage of the idea that it is possible to derive an IC-optimal schedule for a complex DAG by decomposing it into simple components, scheduling each component independently, and then combining the resulting schedules [3].…”

“…In case of DAGs, keeping applications 'ready' means scheduling jobs in a way which maximises the number of jobs that are eligible for mapping to new resources when they appear. An implementation of this strategy was a subject of a series of papers [3,4,5,6]. An algorithm was introduced to provide Internet-Computing (IC) optimal schedule for a large class of DAGs.…”

On clustering DAGs for task-hungry computing platforms

“…[13])and aggressively multi-core and proposed exascale architectures. Ongoing work [4,6,[15][16][17] has been developing IC-scheduling, a master-worker framework for executing static computations that have intertask dependencies (modeled as s) in a way that enhances the rate at which a computation-'s tasks are rendered eligible for execution. IC-scheduling is motivated by the intuition that rendering tasks eligible as fast as possiblea purely -theoretic goalwill enhance the performance of a "task-hungry" platform, no matter what the (instantaneous) distribution of its constituent computer's speeds.…”

“…Both studies suggest that IC-scheduling can accelerate a broad range of computations under a broad range of worker-arrival patterns, by 5% − 30%, or more. IC-scheduling has evolved from the case studies of [16,17] to the algorithmic framework of [4,6,15], and it can now schedule IC-optimallyi.e., in a manner that produces eligible tasks optimally quicklya large repertoire of significant "real" computations; cf. [5].…”

Extending IC-scheduling via the Sweep Algorithm

“…Fig. 2 presents three artificial dags that are also scheduled optimally using the algorithmic framework of [8,26]; these dags illustrate that IC-scheduling does not demand the high degree of structural uniformity observed in the dags of Fig. 1.…”