Multi-GPU Parallelization of the NAS Multi-Zone Parallel Benchmarks

Abstract

GPU-based computing systems have become a widely accepted solution for the high-performance-computing (HPC) domain. GPUs have shown highly competitive performance-per-watt ratios and can exploit an astonishing level of parallelism. However, exploiting the peak performance of such devices is a challenge, mainly due to the combination of two essential aspects of multi-GPU execution. On one hand, the workload should be distributed evenly among the GPUs. On the other hand, communications between GPU devices are costly and should be minimized. Therefore, a trade-of between work-distribution schemes and communication overheads will condition the overall performance of parallel applications run on multi-GPU systems. In this paper we present a multi-GPU implementation of NAS Multi-Zone Parallel Benchmarks (which execution alternate communication and computational phases). We propose several work-distribution strategies that try to evenly distribute the workload among the GPUs. Our evaluations show that performance is highly sensitive to this distribution strategy, as the the communication phases of the applications are heavily affected by the work-distribution schemes applied in computational phases. In particular, we consider Static, Dynamic and Guided schedulers to find a trade-off between both phases to maximize the overall performance. In addition, we compare those schedulers with an optimal scheduler computed offline using IBM CPLEX. On an evaluation environment composed of 2 x IBM Power9 8335-GTH and 4 x GPU NVIDIA V100 (Volta), our multi-GPU parallelization outperforms single-GPU execution from 1.48x to 1.86x (2 GPUs) and from 1.75x to 3.54x (4 GPUs). This paper analyses these improvements in terms of the relationship between the computational and communication phases of the applications as the number of GPUs is increased. We prove that Guided schedulers perform at similar level as optimal schedulers.

Heterogeneous programming using OpenMP and CUDA/HIP for hybrid CPU-GPU scientific applications

“…The basis for the performance evaluation is the comparison between four versions of the benchmarks in the suite: GPU-based and CPU-based non-hybrid versions already studied in previous studies for the NPB-MZ benchmark suite (Duran et al 2005; Gonzalez and Morancho 2020; González and Morancho 2021) and two hybrid executions under two work-distribution schemes. On the one hand, a static scheduling that corresponds to the default and only supported scheduling for hybrid executions in the latest OpenMP specification (OpenMP 5.2).…”