Large-scale performance of a DSL-based multi-block structured-mesh application for Direct Numerical Simulation

Abstract

SBLI (Shock-wave/Boundary-layer Interaction) is a large-scale Computational Fluid Dynamics(CFD) application, developed over 20 years at the University of Southampton and extensively used within the UK Turbulence Consortium. It is capable of performing Direct Numerical Simulations (DNS) or Large Eddy Simulation (LES) of shock-wave/boundarylayer interaction problems over highly detailed multi-block structured mesh geometries. SBLI presents major challenges in data organization and movement that need to be overcome for continued high performance on emerging massively parallel hardware platforms. In this paper we present research in achieving this goal through the OPS embedded domainspecific language. OPS targets the domain of multi-block structured mesh applications. It provides an API embedded in C/C++ and Fortran and makes use of automatic code generation and compilation to produce executables capable of running on a range of parallel hardware systems. The core functionality of SBLI is captured using a new framework called OpenSBLI which enables a developer to declare the partial differential equations using Einstein notation and then automatically carryout discretization and generation of OPS (C/C++) API code. OPS is then used to automatically generate a wide range of parallel implementations. Using this multi-layered abstractions approach we demonstrate how new opportunities for further optimizations can be gained, such as fine-tuning the computation intensity and reducing data movement and apply them automatically. Performance results demonstrate there is no performance loss due to the high-level development strategy with OPS and OpenSBLI, with performance matching or exceeding the hand-tuned original code on all CPU nodes tested. The data movement optimizations provide over 3× speedups on CPU nodes, while GPUs provide 5× speedups over the best performing CPU node. The OPS generated parallel code also demonstrates excellent scalability on nearly 100K cores on a Cray XC30 (ARCHER at EPCC) and on over 4K GPUs on a CrayXK7 (Titan at ORNL).

Shock-Wave/Boundary-Layer Interactions in Transitional Rectangular Duct Flows

“…4 Comparison of time-averaged skin-friction contours on the bottom wall for case A2. Showing (top) the baseline grid resolution and (bottom) a grid coarsened by 25% in all directions the Oxford Parallel Structured (OPS) embedded domain-specific-language (eDSL) (Reguly et al 2014;Mudalige et al 2019). The base code undergoes a source-to-source translation step to a range of computational architectures including CUDA+MPI for multi-GPU clusters.…”

Assessment of Low-Dissipative Shock-Capturing Schemes for the Compressible Taylor–Green Vortex

“…For both platforms the relative runtime was compared to a standard WENO5-JS scheme, to see the increase in computational cost required to achieve low-dissipative shock capturing. Performance of the OPS library used for parallel execution of the code has been reported previously in [30,31], for large-scale DNS on multiple architectures. Additionally, parallel efficiency of the TENO shock-capturing schemes has been reported in [32], for weak-and strong-scaling on multi-GPU configurations.…”

A Data-Centric Directive-Based Framework to Accelerate Out-of-Core Stencil Computation on a GPU

“…Reguly et al [21] presented a cache-blocking tiling technique that efficiently processes large scale stencil code. Their implementations are based on OPS [22]- [24]-a domain specific language (DSL) that requires implementing stencil code based on its syntax. Instead, the PACC framework requires no modification aside from the insertion of directives into serial stencil code, thus reducing programming effort.…”