Studying the impact of synchronization frequency on scheduling tasks with dependencies in heterogeneous systems

Abstract

Many scheduling algorithms have been devised for nested loops with and without dependencies on general heterogeneous distributed systems ([1] and references therein). However, none addressed the case of dynamically computing and allocating chunks of nonindependent tasks to processors. We propose a theoretical model that results in a function that estimates the parallel time of tasks in loops with dependencies on heterogeneous systems. We show that the minimum parallel time is obtained with a synchronization frequency that minimizes the function giving the parallel time. The accuracy of the model is validated through experiments from a practical application. For more details refer to [2].To find the optimal synchronization frequency, we build a theoretical model for heterogeneous dedicated systems, in which workers have different computational powers. Loops with dependencies are efficiently scheduled on heterogeneous systems with selfscheduling algorithms [1]. The self-scheduling algorithms are based on the master-worker model. The master assigns work to workers upon request. Due to the data dependencies, applying self-scheduling algorithms to loops with dependencies yields a pipelined parallel execution. In the case of one master and N P workers, each assignment round corresponds to a pipeline with N P stages. Our approach assumes that the nested loop is represented in Cartesian space with at least 2 dimensions. One dimension is partitioned by the master into chunks according to a self-scheduling algorithm. In a pipeline organization, each worker synchronizes with its neighbors. Thus, synchronization points are inserted along the other dimension. A synchronization interval, denoted by h, represents the number of elements in the index space along the synchronization dimension. Data produced at the end of one pipeline are fed to the next pipeline. It is obvious that the synchronization frequency plays an important role in the total parallel time. Frequent synchronization implies excessive communication, whereas infrequent synchronization may limit the parallelism.In order to estimate the theoretical parallel time on a heterogeneous system for the case of multiple assignment rounds (pipelines), i.e., the number of processors is smaller than the total number of chunks, we assume that a problem of the original index space size can be decomposed into p subproblems (pipelines) of (equal) size in which each processor is assigned one chunk. Thus, one subproblem corresponds to one assignment round. These subproblems are inter-dependent in the sense that (part of) the data produced by one subproblem are consumed by the next subproblem. Upon completion of one subproblem, the processor assigned the last chunk of the subproblem transmits (in a single message) all necessary data to the processor assigned the first chunk of the next subproblem. The time to complete this data transfer, represents the time to send and receive a data packet of size equal to the size of the synchronization dimension.Hence, the theoretical parallel ...

Tiling and Scheduling of Three-level Perfectly Nested Loops with Dependencies on Heterogeneous Systems

“…There are many approaches that attempt to determine the near-optimal tile size in homogeneous platforms such as analytical, auto-tuning and evolutionary approaches [25,26]. In heterogeneous platforms, tile size determined by the computational power awareness of the processing nodes [21,27,28]. The 3D tiling of the nested loop with dependencies for heterogeneous systems has not been given enough attention so far.…”

Algorithms design for the parallelization of nested loops

“…• A theoretical model for determining the optimal synchronization/communication fre quency between processors for the execution of DOACROSS loops on heterogeneous systems [CRA+07a].…”

“…The theoretical model described in this section is one of the first attempts to determine the optimal synchronization frequency for DOACROSS loops on heterogeneous systems. It appeared in [CRA+07a] and its extended version [CRA+07b] is currently under review. To find the optimal synchronization frequency, we first build a theoretical model for predicting the minimum parallel time for the simpler case of homogeneous dedicated systems.…”