Dynamic Load Balancing for Unstructured Meshes on Space-Filling Curves

Abstract

Load imbalance is an important impediment on the path towards higher degrees of parallelism-especially for engineering codes with their highly unstructured problem domains. In particular, when load conditions change dynamically, efficient mesh partitioning becomes an indispensable ingredient of scalable design. However, popular graph-based methods such as those used by ParMetis require global knowledge, which effectively limits the problem size on distributed-memory machines. On such architectures, space-filling curves (SFCs) offer a memory-efficient alternative and many sophisticated schemes have already been proposed. In this paper, we present a simple strategy based on SFCs that is custom-tailored to the needs of static meshes with dynamically changing computational load. Exploiting the properties of this class of problems, it is not only easy to implement but also reduces memory requirements substantially. Moreover, exclusively relying on MPI collective operations, our load-balancing scheme also offers portable performance across a broad range of HPC systems. Experimental evaluation shows excellent scaling behavior for up to 16,384 cores on a Nehalem-Infiniband system and up to 294,912 processes on a Blue Gene/P system.

Dynamic load balancing for a mesh‐based scientific application

“…Another essential piece in load balancing is to determine when to trigger a load balancing step. There are many iterative applications which are widely used in scientific applications such as atmospheric simulation, molecular dynamics, and turbulent streams and shocks simulation . It is intuitive to invoke a load balancing at a fixed time interval, for example, every 500 time steps.…”

“…There are many iterative applications which are widely used in scientific applications such as atmospheric simulation, 19 molecular dynamics, 11 and turbulent streams and shocks simulation. 22 It is intuitive to invoke a load balancing at a fixed time interval, for example, every 500 time steps. However, whether this invocation is profitable or not should be investigated since performing load balancing involves an overhead.…”

Laika

“…As using recursive space-filling curves to reorder input data for better locality is an established practice [16,17,27,33], we first consider the cache miss rate of the traversal of a reordered random cube graph -n vertices uniformly randomly distributed in the unit cube and connected to all other vertices within a distance r . The best prior upper bound for misses of an M-vertex cache during a traversal of such a graph is O(M −1/4 ) [29].…”

Scalable high-quality 1D partitioning

“…Many scientific simulations exhibit workload variations due to adaptive spatial grids [3] or adaptive time stepping techniques [4], [5]. An additional source of workload variations are descriptions of physical or chemical phenomena, whose runtime depend locally on variables of the model, like detailed atmospheric simulations [6], [7] and particle simulations [8], [9].…”

“…It is applied for scalable adaptive mesh refinement [3] and dynamic load balancing of a fixed number of tasks with varying workload [4], [8]. In general, SFCs provide a fast mapping from n-dimensional to one-dimensional space that preserves spatial locality.…”