Dynamic load balancing with adaptive factoring methods in scientific applications

Abstract

To improve the performance of scientific applications with parallel loops, dynamic loop scheduling methods have been proposed. Such methods address performance degradations due to load imbalance caused by predictable phenomena like nonuniform data distribution or algorithmic variance, and unpredictable phenomena such as data access latency or operating system interference. In particular, methods such as factoring, weighted factoring, adaptive weighted factoring, and adaptive factoring have been developed based on a probabilistic analysis of parallel loop iterates with variable running times. These methods have been successfully implemented in a number of applications such as: N-Body and Monte Carlo simulations, computational fluid dynamics, and radar signal processing.The focus of this paper is on adaptive weighted factoring (AWF), a method that was designed for scheduling parallel loops in time-stepping scientific applications. The main contribution of the paper is to relax the time-stepping requirement, a modification that allows the AWF to be used in any application with a parallel loop. The modification further allows the AWF to adapt to load imbalance that may occur during loop execution. Results of experiments to compare the performance of the modified AWF with the performance of the other loop scheduling methods in the context of three nontrivial applications reveal that the performance of the modified method is

Dynamic Load Balancing of Parallel Computational Iterative Routines on Platforms with Memory Heterogeneity

“…These strategies are restricted to applications with pre-determined workload and cannot be applied to such iterative routines as adaptive mesh refinement [7], for which the amount of computation data grows unpredictably. Dynamic algorithms [8,9,10,11,12] do not require a priori information and can be used with a wider class of parallel applications. In addition, dynamic algorithms can be deployed on non-dedicated platforms.…”

“…In non-centralised algorithms [11,12], load is migrated locally between neighbouring processors, while in centralised ones [4,5,6,8,9,10], load is distributed based on global load information. Non-centralized algorithms are slower to converge.…”

“…Task queue algorithms [9,10] distribute tasks. They target parallel routines consisting of independent tasks and schedule them on sharedmemory platforms.…”

“…It has been shown that the task queue model implemented in [10] can outperform the model [9] because decisions are based on adaptive speed measurements rather then single speed measurements. The algorithm presented in this paper also applies an adaptive performance model, but in such a way that it is applicable to scientific computational iterative routines.…”

Adaptive statistical scheduling of divisible workloads in heterogeneous systems

“…Safe self-scheduling augments such an approach with expected execution times, obtained through profiling or previous runs, to determine a variable number of operations (Liu et al 1994). Adaptive factoring methods employ historical execution times of certain loop iterations, on a per processor basis, to adjust the number of operations delegated to a processor (Cariño and Banicescu 2008). These methods arguably supersede the traditional batch-oriented factoring, where a processing batch is determined through a fixed ratio of pending iterations and then divided equally among participating processors.…”