On the Benefits of Anticipating Load Imbalance for Performance Optimization of Parallel Applications

Abstract

In parallel iterative applications, computational efficiency is essential for addressing large problems. Load imbalance is one of the major performance degradation factors of parallel applications. Therefore, distributing, cleverly, and as evenly as possible, the workload among processing elements (PE) maximizes application performance. So far, the standard load balancing method consists in distributing the workload evenly between PEs and, when load imbalance appears, redistributing the extra load from overloaded PEs to underloaded PEs. However, this does not anticipate the load imbalance growth that may continue during the next iterations. In this paper, we present a first step toward a novel philosophy of load balancing that unloads the PEs that will be overloaded in the near future to let the application rebalance itself via its own dynamics. Herein, we present a formal definition of our new approach using a simple mathematical model and discuss its advantages compared to the standard load balancing method. In addition to the theoretical study, we apply our method to an application that reproduces the computation of a fluid model with non-uniform erosion. The performance validates the benefit of anticipating load imbalance. We observed up to 16% performance improvement compared to the standard load balancing method.

Towards Informed Partitioning for Load Balancing: a Proof-of-Concept

“…In a previous paper, we developed a load balancing paradigm in which we used information about the workload increase rate to "underload" the processing elements that are currently overloading, such that they will catch up due to their imbalance "momentum" [26]. This paradigm, combined with a simple stripe load balancing scheme, allowed us to improve the performance of a simulation of stochastic rock erosion application by up to 16% compared to the classical HSFC algorithm from Zoltan.…”

“…In the bottom left, classical load balancing methods does not use load imbalance anticipation nor informed partitioning to increase the partition lifetime. In the top left corner, ULBA [26] anticipates the imbalance but does not adapt the partitions to it. Finally, in the bottom right corner, NoRCB (current paper) adapts the partition to reduce the load imbalance growth (ideally to suppress it).…”

Optimal Load Balancing and Assessment of Existing Load Balancing Criteria

“…Moreover, due to the complexity of modern algorithms and the lack of "plug and play" libraries, scientists often use the most famous load balancing techniques, which may not be optimal for their problem. In addition, we pointed out in a previous work that researchers should not select a load balancing technique only based on its capability to correct imbalance but also during how many iterations it keeps a low level of imbalance [10]. This further increases the difficulty to select the most optimal technique.…”

Accurate load balancing accelerates Lagrangian simulation of water ages on distributed, multi-GPU platforms

“…Load balancing (LB) schemes have been used to overcome these parallelization challenges discussed above. Other disciplines using particle tracking, such as the molecular dynamics (MD) and the smoothed particle hydrodynamics (SPH) (Boulmier, Raynaud, Abdennadher, & Chopard, 2019;Egorova, Dyachkov, Parshikov, & Zhakhovsky, 2019;Eibl & Rüde, 2019;Fattebert, Richards, & Glosli, 2012;Furuichi & Nishiura, 2017;Kunaseth et al, 2013) have presented LB schemes that greatly improved parallel simulation performance. However, LB has not been applied to hydrologic modeling based on particle tracking; even in the studies using MPI through DDC mentioned above.…”