HPCG and HPGMG benchmark tests on multiple program, multiple data (MPMD) mode on Blue Waters—A Cray XE6/XK7 hybrid system

Abstract

SummaryThe high-performance conjugate gradients (HPCG) and high-performance geometric multi-grid (HPGMG) benchmarks are alternatives to the traditional LINPACK benchmark (HPL) in measuring the performance of modern HPC platforms. We performed HPCG and HPGMG benchmark tests on a Cray XE6/XK7 hybrid supercomputer, Blue Waters at National Center for Supercomputing Applications (NCSA). The benchmarks were tested on CPU-based and GPUenabled nodes separately, and then we analyzed characteristic parameters that affect their performance. Based on our analyses, we performed HPCG and HPGMG runs in multiple program, multiple data (MPMD) mode in Cray Linux Environment in order to measure their hybrid performance on both CPU-based and GPU-enabled nodes. We observed and analyzed several performance issues during those tests. Based on lessons learned from this study, we provide recommendations about how to optimize science applications on modern hybrid HPC platforms. KEYWORDSGPU, heterogeneous computing, HPC benchmark, hybrid HPC platforms, MPMD mode | INTRODUCTIONSince 1979, LINPACK benchmark (HPL) 1 has been used to measure the performance of HPC systems and the TOP500 list has been accordingly updated to rank the world's most powerful supercomputer. However, there have been many opinions criticizing whether HPL is a good metric to measure modern HPC performance or not. [2][3][4][5] One of the main criticisms is its low memory to flop (Byte/Flop) ratio. Byte/Flop ratios of many applications in molecular dynamics, weather forecasting, astrophysics, particle physics, structural analysis, and fluid dynamics are between 10 −1 and 10, while the ratio of HPL benchmark is less than 10 −3 . Since 2014, HPC community has shared the high-performance conjugate gradients (HPCG) 3,6 and high-performance geometric multi-grid (HPGMG) 4,7 benchmark results in order to represent modern HPC performance for many science and engineering HPC applications, at least, in terms of Byte/Flop ratios.This study is an extension of our continuing effort 8 to find out appropriate benchmarks for modern HPCs. In this study, we employed HPCG and HPGMG benchmarks to measure the performance of the Blue Waters system located at National Center for Supercomputing Application (NCSA). Blue Waters 9,10 is a Cray XE6/XK7 hybrid supercomputer with 22 640 CPU-based XE6 nodes and 4228 GPU-enabled XK7 nodes. The XE6 dual-socket nodes are populated with 2 AMD Interlargos model 6276 CPU processors with a nominal clock speed of at least 2.3 GHz and 64 GB of physical memory, while the XK7 accelerator nodes are equipped with one Interlagos model 6276 CPU processor and one NVIDIA GK110 "Kepler" accelerator K20X with 32 GB of CPU memory and 6 GB of GPU memory. We first tested HPCG and HPGMG on dual-socket CPU-based nodes and single-socket GPU-enabled nodes separately. After analyzing numerous configurations for the optimal performance, we moved on multiple program, multiple data (MPMD) runs to evaluate the performance on CPU-based XE6 nodes and GPU-enabled XK7 nodes together. ...

Foreword to the special issue of the Cray User Group (CUG 2017)

“…The articles in this special issue touch on many of the new developments in both the hardware and software architecture. In addition to the papers from the themes outlined above, several papers highlighted topics in systems support, application optimization, and programming environments . We hope that readers will benefit from the knowledge and insight into operating large‐scale Cray machines contained in this issue.…”

Roofline analysis with Cray performance analysis tools (CrayPat) and roofline‐based performance projections for a future architecture

“…We first provide roofline analysis plots for 2D/3D stencil kernels discussed in the previous section. Second, we report CrayPat‐based roofline analysis results for High‐Performance Conjugate Gradients (HPCG) and High‐Performance Geometric Multigrid (HPGMG) benchmarks that represent multiple sets of kernels employed by many science and engineering applications . Last, we introduce roofline analysis plots for sparse linear algebra libraries with multiple non‐symmetric matrices from a computational fluid dynamics (CFD) code .…”

“…Second, we report CrayPat-based roofline analysis results for High-Performance Conjugate Gradients (HPCG) and High-Performance Geometric Multigrid (HPGMG) benchmarks that represent multiple sets of kernels employed by many science and engineering applications. 6,13,14 Last, we introduce roofline analysis plots for sparse linear algebra libraries with multiple non-symmetric matrices from a computational fluid dynamics (CFD) code. 4 We employed PETSc 15 for parallel sparse iterative solvers, and MUMPS 16 and SuperLU 17 for parallel sparse direct solvers.…”

“…The performance of many HPC applications is actually bounded by other components of modern HPC systems: latency and bandwidth of cache, memory and network. For that reason, we have made continuous efforts to develop comprehensive and accurate benchmarks for modern HPC systems by employing prevailing kernels used by modern HPC codes, and popular HPC applications engaged by many domain‐specific science groups …”

“…For that reason, we have made continuous efforts to develop comprehensive and accurate benchmarks for modern HPC systems by employing prevailing kernels used by modern HPC codes, and popular HPC applications engaged by many domain-specific science groups. [4][5][6] The roofline analysis model 7 is a powerful tool for HPC code developers since it provides visually intuitive plots for the performance of their applications and kernels in terms of computational intensity (CI) and flop-rate. Intel ® Advisor started providing a useful cache-aware roofline analysis model 8 in a handy way since its version 2017 update 2; however, it is not widely compatible with other compilers (eg, PGI, CRAY, GNU) and other processor-architectures (eg, AMD, ARM, GPUs).…”