On the Parallelization of a Cache-Optimal Iterative Solver for PDEs Based on Hierarchical Data Structures and Space-Filling Curves

Abstract

Abstract. Competitive numerical simulation codes solving partial differential equations have to tap the full potential of both modern numerical methods -like multi-grid and adaptive grid refinement -and available computing resources. In general, these two are rival tasks. Typically, hierarchical data structures resulting from multigrid and adaptive grid refinement impede efficient usage of modern memory architectures on the one hand and complicate the efficient parallelization on the other hand due to scattered data for coarse-level-points and unbalanced data trees. In our previous work, we managed to bring together high performance aspects in numerics as well as in hardware usage in a very satisfying way. The key to this success was to integrate space-filling curves consequently not only in the programs flow control but also in the construction of data structures which are processed linearly even for hierarchical multilevel data. In this paper, we present first results on the second challenge, namely the efficient parallelization of algorithms working on hierarchical data. It shows that with the same algorithms as desribed above, the two main demands on good parellel programs can be fulfilled in a natural way, too: The balanced data partitioning can be done quite easily and cheaply by cutting the queue of data linearized along the space-filling curve into equal pieces. Furtheron, this partitioning is quasi-optimal regarding the amount of communication. Thus, we will end up with a code that overcomes the quandary between hierarchical data and efficient memory usage and parallelization in a very natural way by a very deep integration of space-filling-curves in the underlying algorithm.

A Parallel Adaptive Cartesian PDE Solver Using Space–Filling Curves

“…Space-filling curves [15] are well known to simplify a lot of different tasks, due to their good locality properties ( [3,5,6,7,8,10,12,13] e.g.). Their recursive, selfsimilar definition implies a depth-first traversal of the corresponding cell tree and, therefore, an enumeration of the cells of all levels.…”

“…Our idea is to use stacks, since they meet the resulting requirements: put a record (vertex) on the top of a stack after using it the first time, and pop a record from the stack when it is needed the second time. In addition to the left and right stack, one has to add a third idea, the stack colouring, within a hierarchical grid as pointed out first by [6], to avoid access conflicts due to the top-down bottom-up steps of the traverse in the generating system since there might be more than one degree of freedom per vertex.…”

“…We want to address this problem and, in the following, will derive a traversal and data management algorithm working on adaptive cartesian grids alike [7,12]. This algorithm then is parallelised using a domain decomposition approach based on [6]. Although the results are presented for a three-dimensional Poisson problem on an a priori refined grid only, we are able to solve any d-dimensional problem that can be discretised by a 3 d -point stencil.…”

Towards multi-phase flow simulations in the PDE framework Peano

“…The parallelisation of Peano follows a domain decomposition approach based on the ordering of grid cells along the corresponding discrete iterate of the Peano curve. Simply spoken, the queue of grid cells resulting from the ordering along the respective iterate of the Peano curve is cut into equal pieces to achieve a balanced partitioning [24,29]. Parallelisation concepts based on space-filling curves have become more and more popular in the recent years and have successfully been implemented by different groups (see for example [11,23,26,46,56] and citations therein).…”

A Modular and Efficient Simulation Environment for Fluid-Structure Interactions with Large Domain Deformation

“…Hereby, the principle idea behind is to queue up the grid cells (on all refinement levels) in the order prescribed by the space-filling curve and, subsequently, to cut this queue into equal pieces. Peano adopts this principle [29,24,22]. In the last two years, the original concept has been modified to a more level-oriented approach which has the advantage that already the domain partitioning itself can be done in parallel [36,7,31].…”