Load Balancing for Regular Meshes on SMPs with MPI

Abstract

Domain decomposition for regular meshes on parallel computers has traditionally been performed by attempting to exactly partition the work among the available processors (now cores). However, these strategies often do not consider the inherent system noise which can hinder MPI application scalability to emerging peta-scale machines with 10000+ nodes. In this work, we suggest a solution that uses a tunable hybrid static/dynamic scheduling strategy that can be incorporated into current MPI implementations of mesh codes. By applying this strategy to a 3D jacobi algorithm, we achieve performance gains of at least 16% for 64 SMP nodes.

Weighted locality-sensitive scheduling for mitigating noise on multi-core clusters

“…The above characterization confirms the benefits of the solution proposed in [6]. When a platform has several different noise events of different lengths, a dynamic scheduling strategy with an assortment of task granularities can be used.…”

“…Our initial work shows that the performance improvement of an application is relatively unnoticeable when running on a small number of nodes of a cluster, but becomes much more dramatic as we scale the application to a large number of nodes [6].…”

“…However, a preceding study that we did shows that through careful tuning to keep the cost of dynamic scheduling low while trying to minimize small-scale load imbalances within a node, we can achieve more predictable performance within a multi-core processor; this ultimately allows for better scalability as we run on a larger number of nodes [6]. The amount of dynamic scheduling we allow is proportional to the duration of the characteristic system noise of a machine.…”

Hybrid Static/dynamic Scheduling for Already Optimized Dense Matrix Factorization

“…V. Kale et al suggested a hybrid static/dynamic approach in [16] that can be incorporated into current MPI implementations of structured grid codes to improve the load balancing of the initial static decompositions. This work embraces the fundamental principles advocated in [16], and applies it in the context of dense matrix factorizations. Xue et al introduced an approach in [24] that improves the data locality when executing loop iterations in codes.…”

“…In order to be scalable for future high-performance clusters(i.e. exascale), the code running within a node of a cluster must be tuned such that it achieves not simply "high-performance", but also "performance consistency" [14], [16]. Such static tuning techniques provide few guarantees on performance consistency.…”