Performance Analysis of Irregular Collective Communication with the Crystal Router Algorithm

Schliephake, Michael; Laure, Erwin

doi:10.1007/978-3-319-15976-8_10

Cited by 3 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The crystal router algorithm performs the nearest neighbor collective via recursive hypercube folding. The algorithm is described in full by Lamb et al (1988), and some performance results of this algorithm’s use in Nek5000 were presented by Schliephake and Laure (2014). For a P process grid, the algorithm can be summarized as follows: divide the grid in half and pair each process p l in the lower half with a distinct process p h in the upper half.…”

Section: The Nekbone Benchmarkmentioning

confidence: 99%

HipBone: A performance-portable graphics processing unit-accelerated C++ version of the NekBone benchmark

Chalmers

Mishra

McDougall

et al. 2023

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

We present hipBone, an open-source performance-portable proxy application for the Nek5000 (and NekRS) computational fluid dynamics applications. HipBone is a fully GPU-accelerated C++ implementation of the original NekBone CPU proxy application with several novel algorithmic and implementation improvements which optimize its performance on modern fine-grain parallel GPU accelerators. Our optimizations include a conversion to store the degrees of freedom of the problem in assembled form in order to reduce the amount of data moved during the main iteration and a portable implementation of the main Poisson operator kernel. We demonstrate near-roofline performance of the operator kernel on three different modern GPU accelerators from two different vendors. We present a novel algorithm for splitting the application of the Poisson operator on GPUs which aggressively hides MPI communication required for both halo exchange and assembly. Our implementation of nearest-neighbor MPI communication then leverages several different routing algorithms and GPU-Direct RDMA capabilities, when available, which improves scalability of the benchmark. We demonstrate the performance of hipBone on three different clusters housed at Oak Ridge National Laboratory, namely, the Summit supercomputer and the Frontier early-access clusters, Spock and Crusher. Our tests demonstrate both portability across different clusters and very good scaling efficiency, especially on large problems.

show abstract

Section: The Nekbone Benchmarkmentioning

confidence: 99%

HipBone: A performance-portable graphics processing unit-accelerated C++ version of the NekBone benchmark

Chalmers

Mishra

McDougall

et al. 2023

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…This is similar to the Global Segment Map (GSMap) in MCT, which in contrast is stored in every processor, leading to O(N x ) memory requirements. The parallel communication infrastructure in MOAB is heavily leveraged (Tautges et al, 2012) to utilize the scalable crystal router algorithm (Fox et al, 1989;Schliephake and Laure, 2015) in order to scalably communicate the covering cells to different processors. This parallel mesh infrastructure in MOAB provides the necessary algorithmic tools for optimally executing online remapping strategies, so that MCT in E3SM can be replaced with a MOAB-based coupler.…”

Section: Algorithmic Approachmentioning

confidence: 99%

Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x)

et al. 2020

View full text Add to dashboard Cite

Abstract. One of the fundamental factors contributing to the spatiotemporal inaccuracy in climate modeling is the mapping of solution field data between different discretizations and numerical grids used in the coupled component models. The typical climate computational workflow involves evaluation and serialization of the remapping weights during the preprocessing step, which is then consumed by the coupled driver infrastructure during simulation to compute field projections. Tools like Earth System Modeling Framework (ESMF) (Hill et al., 2004) and TempestRemap (Ullrich et al., 2013) offer capability to generate conservative remapping weights, while the Model Coupling Toolkit (MCT) (Larson et al., 2001) that is utilized in many production climate models exposes functionality to make use of the operators to solve the coupled problem. However, such multistep processes present several hurdles in terms of the scientific workflow and impede research productivity. In order to overcome these limitations, we present a fully integrated infrastructure based on the Mesh Oriented datABase (MOAB) (Tautges et al., 2004; Mahadevan et al., 2015) library, which allows for a complete description of the numerical grids and solution data used in each submodel. Through a scalable advancing-front intersection algorithm, the supermesh of the source and target grids are computed, which is then used to assemble the high-order, conservative, and monotonicity-preserving remapping weights between discretization specifications. The Fortran-compatible interfaces in MOAB are utilized to directly link the submodels in the Energy Exascale Earth System Model (E3SM) to enable online remapping strategies in order to simplify the coupled workflow process. We demonstrate the superior computational efficiency of the remapping algorithms in comparison with other state-of-the-science tools and present strong scaling results on large-scale machines for computing remapping weights between the spectral element atmosphere and finite volume discretizations on the polygonal ocean grids.

show abstract

HipBone: A performance-portable GPU-accelerated C++ version of the NekBone benchmark

Chalmers¹,

Mishra²,

McDougall³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present hipBone, an open source performance-portable proxy application for the Nek5000 (and NekRS) CFD applications. HipBone is a fully GPU-accelerated C++ implementation of the original NekBone CPU proxy application with several novel algorithmic and implementation improvements which optimize its performance on modern finegrain parallel GPU accelerators. Our optimizations include a conversion to store the degrees of freedom of the problem in assembled form in order to reduce the amount of data moved during the main iteration and a portable implementation of the main Poisson operator kernel. We demonstrate near-roofline performance of the operator kernel on three different modern GPU accelerators from two different vendors. We present a novel algorithm for splitting the application of the Poisson operator on GPUs which aggressively hides MPI communication required for both halo exchange and assembly. Our implementation of nearest-neighbor MPI communication then leverages several different routing algorithms and GPU-Direct RDMA capabilities, when available, which improves scalability of the benchmark. We demonstrate the performance of hipBone on three different clusters housed at Oak Ridge National Laboratory, namely the Summit supercomputer and the Frontier early-access clusters, Spock and Crusher. Our tests demonstrate both portability across different clusters and very good scaling efficiency, especially on large problems.

show abstract

Performance Analysis of Irregular Collective Communication with the Crystal Router Algorithm

Cited by 3 publications

References 10 publications

HipBone: A performance-portable graphics processing unit-accelerated C++ version of the NekBone benchmark

HipBone: A performance-portable graphics processing unit-accelerated C++ version of the NekBone benchmark

Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x)

HipBone: A performance-portable GPU-accelerated C++ version of the NekBone benchmark

Contact Info

Product

Resources

About