An MPI Halo-Cell Implementation for Zero-Copy Abstraction

Interoperability strategies for GASPI and MPI in large-scale scientific applications

“…The idea for a zero-copy framework for ghost cell exchanges has been discussed in Besnard et al (2015). Similar to our work, the authors strip away the node-local copying of ghost cell data and instead directly access the inner ghost cell data structures of node-local (and neighboring) ranks.…”

“…This implementation relies on a “threads as processes” (MPC) MPI implementation, while we start from a more general approach; we directly use MPI shared memory across multiple processes and run the solver (wherever we subsequently need to exchange corresponding data) in these shared windows. Our work also conceptually extends the work in Besnard et al (2015), as we implement a model where communication (not just computation) is visible for all processes in the shared window. Furthermore, the implementation for the pipelined Allreduce, which is presented in Section 4.4, would not be feasible with the approach from Besnard et al (2015), as all processes require access to node-local communication to optimally sustain the pipeline.…”

“…Our work also conceptually extends the work in Besnard et al (2015), as we implement a model where communication (not just computation) is visible for all processes in the shared window. Furthermore, the implementation for the pipelined Allreduce, which is presented in Section 4.4, would not be feasible with the approach from Besnard et al (2015), as all processes require access to node-local communication to optimally sustain the pipeline.…”

Direct Numerical Simulation of Stratified Turbulent Flows and Passive Tracer Transport on HPC Systems: Comparison of CPU Architectures

“…Common optimizations for improving scaling on HPC systems include options for combining MPI data exchanges for a number of arrays (e.g., vector or tensor compo-nents) or increasing the width of the grid halo region (see, e.g. [3]) for reducing latency of MPI communications. The latter allows to reduce the number of calls to MPI functions but at the cost of additional computational overhead, which may be negligible when the size of the problem on MPI-process is comparatively small.…”

Hardware Locality-Aware Partitioning and Dynamic Load-Balancing of Unstructured Meshes for Large-Scale Scientific Applications

“…A more intrusive approach is to get rid of halo intranode halo buffers in the users code and directly expose the neighbouring intranode processor's memory. So the intranode transfer becomes simple direct memory-to-memory copy of halo data without any intermediate buffers (zero-copy ) [5].…”