Towards the optimal synchronization granularity for dynamic scheduling of pipelined computations on heterogeneous computing systems

Abstract

Loops are the richest source of parallelism in scientific applications. A large number of loop scheduling schemes have therefore been devised for loops with and without data dependencies (modeled as dependence distance vectors) on heterogeneous clusters. The loops with data dependencies require synchronization via cross-node communication. Synchronization requires fine-tuning to overcome the communication overhead and to yield the best possible overall performance. In this paper, a theoretical model is presented to determine the granularity of synchronization that minimizes the parallel execution time of loops with data dependencies when these are parallelized on heterogeneous systems using dynamic self-scheduling algorithms. New formulas are proposed for estimating the total number of scheduling steps when a threshold for the minimum work assigned to a processor is assumed. The proposed model uses these formulas to determine the synchronization granularity that minimizes the estimated parallel execution time. The accuracy of the proposed model is verified and validated via extensive experiments on a heterogeneous computing system. The results show that the theoretically optimal synchronization granularity, as determined by the proposed model, is very close to the experimentally observed optimal synchronization granularity, with no deviation in the best case, and within 38.4% in the worst case.

Tiling and Scheduling of Three-level Perfectly Nested Loops with Dependencies on Heterogeneous Systems

“…Comparison of 3D and 2D Tiling. As mentioned in the related work, the proposed methods in [21,33,40] could find the near-optimal partitioning of 3-nested loop with dependencies for homogeneous/heterogeneous computing systems. It targets two loops of the nested loop and considers the outer loop as synchronization dimension and another loop as scheduling dimension.…”

“…In distributed-memory parallel systems, communication and synchronization overhead between the nodes are the important reasons of the performance degradation when running dependence loops. So, we use coarse-grain pipeline parallelism to balance trade-offs between parallelization, communication and synchronization overhead [20,33].…”

“…Andronikos et al [21,33,40] claimed that the problem of finding the optimal partitioning of nested loops [21,33,40] on homogeneous and heterogeneous systems, respectively. The tiles with the same number can be executed simultaneously.…”

“…In this section, we propose an approach to 3D tiling and scheduling of three-level perfectly nested loops with dependencies on heterogeneous systems. In the paper, we use the notation in [21,33], indicated in Table 3.1. Algorithm 1 outlines the main steps of proposed method.…”

Scalable Loop Self-Scheduling Schemes for Large-Scale Clusters and Cloud Systems

“…A two-phase scheme is proposed to solve parallel regular loop scheduling problem in heterogeneous grid computing environments in [16]. In [17][18][19] new results are presented for loops with dependencies. Recent research results [20,21] have been reported for designing loop self-scheduling methods for grids.…”