Accelerating communication for parallel programming models on GPU systems

Choi, Jaemin; Fink, Zane; White, Sam; Bhat, Nitin; Richards, David F.; Kalé, Laxmikant V.

doi:10.1016/j.parco.2022.102969

Cited by 1 publication

(1 citation statement)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Explicit multi‐node GPU programming using CUDA and MPI can benefit from CUDA‐aware MPI implementations allowing usage of GPU memory pointers for GPU‐GPU communication on various nodes. CUDA inter‐process communication (IPC) allows GPU‐GPU communication that can cross the process boundary 14 with optimizations for intra‐node MPI communication for multi‐GPU nodes shown in paper 15.…”

Section: Related Workmentioning

confidence: 99%

A multithreaded CUDA and OpenMP based power‐aware programming framework for multi‐node GPU systems

Czarnul

2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryIn the article, we have proposed a framework that allows programming a parallel application for a multi‐node system, with one or more graphical processing units (GPUs) per node, using an OpenMP+extended CUDA API. OpenMP is used for launching threads responsible for management of particular GPUs and extended CUDA calls allow to transfer data and launch kernels on local and remote GPUs. The framework hides inter‐node MPI communication from the programmer. For optimization, the implementation takes advantage of the MPI_THREAD_MULTIPLE mode allowing: multiple threads handling distinct GPUs as well as overlapping communication and computations transparently using multiple CUDA streams. The solution allows data parallelization across available GPUs in order to minimize execution time and supports a power‐aware mode in which GPUs are automatically selected for computations using a greedy approach in order not to exceed an imposed power limit. We have implemented and benchmarked three parallel applications including: finding the largest divisors; verification of the Collatz conjecture; finding patterns in vectors. These were tested on three various systems: a GPU cluster with 16 nodes, each with NVIDIA GTX 1060 GPU; a powerful 2‐node system—one node with 8 NVIDIA Quadro RTX 6000 GPUs, the second with 4 NVIDIA Quadro RTX 5000 GPUs; a heterogeneous environment with one node with 2 NVIDIA RTX 2080 and 2 nodes with NVIDIA GTX 1060 GPUs. We demonstrated effectiveness of the framework through execution times versus power caps within ranges of 100–1400 W, 250–3000 W, and 125–600 W for these systems respectively as well as gains from using two versus one CUDA streams per GPU. Finally, we have shown that for the testbed applications the solution allows to obtain high speed‐ups between 89.3% and 97.4% of the theoretically assessed ideal ones, for 16 nodes and 2 CUDA streams, demonstrating very good parallel efficiency.

show abstract

Section: Related Workmentioning

confidence: 99%

A multithreaded CUDA and OpenMP based power‐aware programming framework for multi‐node GPU systems

Czarnul

2023

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

Accelerating communication for parallel programming models on GPU systems

Cited by 1 publication

References 2 publications

A multithreaded CUDA and OpenMP based power‐aware programming framework for multi‐node GPU systems

A multithreaded CUDA and OpenMP based power‐aware programming framework for multi‐node GPU systems

Contact Info

Product

Resources

About