Real Asynchronous MPI Communication in Hybrid Codes through OpenMP Communication Tasks

Buettner, David; Acquaviva, Jean-Thomas; Weidendorfer, Josef

doi:10.1109/icpads.2013.39

Cited by 7 publications

(10 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We therefore implemented a progress task (similar to Reference 32) which uses MPI_Testsome on the request manager's request queue to make progress on outstanding MPI requests. In line with the polling, the task is started prior to the first time step and reschedules itself.…”

Section: Methodsmentioning

confidence: 99%

Lightweight task offloading exploiting MPI wait times for parallel adaptive mesh refinement

Samfass

Weinzierl

Charrier

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Balancing the workload of sophisticated simulations is inherently difficult, since we have to balance both computational workload and memory footprint over meshes that can change any time or yield unpredictable cost per mesh entity, while modern supercomputers and their interconnects start to exhibit fluctuating performance. We propose a novel lightweight balancing technique for MPI+X to accompany traditional, prediction-based load balancing. It is a reactive diffusion approach that uses online measurements of MPI idle time to migrate tasks temporarily from overloaded to underemployed ranks. Tasks are deployed to ranks which otherwise would wait, processed with high priority, and made available to the overloaded ranks again. This migration is nonpersistent. Our approach hijacks idle time to do meaningful work and is totally nonblocking, asynchronous and distributed without a global data view. Tests with a seismic simulation code developed in the ExaHyPE engine uncover the method's potential. We found speed-ups of up to 2-3 for ill-balanced scenarios without logical modifications of the code base and show that the strategy is capable to react quickly to temporarily changing workload or node performance. K E Y W O R D S adaptive mesh refinement, MPI+X, reactive load balancing, task-based parallelism 1 INTRODUCTION Load balancing that decomposes work prior to a certain compute phase-a time step or iteration of an equation system solver-is doomed to underperform in many sophisticated simulation codes. There are multiple reasons for this: The clock frequency of processors changes over runtime, 1-3 the network speed is subject to noise due to other applications 4,5 or IO, and task-based multicore parallelization (MPI+X) tends to yield fluttering throughput due to effects of the memory hierarchy, 6 work stealing and nondeterminism in the MPI progression. While this list is not comprehensive, notably modern numerics drive the nonpredictability: They build atop of dynamic adaptive mesh refinement (AMR) that changes the mesh throughout a time step or mesh sweep, 7 combine different physical models, 7-9 or solve nonlinear equation systems with iterative solvers in substeps. 10 It becomes hard or even impossible to predict a step's computational load. As adjusting parallel partitions and respective data migration is often costly, many AMR codes consequently repartition only every 10th or 100th time step and tolerate certain load imbalances in-between. We propose a novel, lightweight load redistribution scheme that acts on top of traditional load balancing. It, first, assumes that parts of the underlying simulation code are phrased in terms of many expensive tasks. It, second, assumes that good AMR codes manage to hide data exchange behind computations yet cannot keep all cores busy all the time. In every solver step, some cores on some ranks have to wait for MPI data to drop in. Our idea is to offload tasks from overbooked to waiting ranks to make these work productively rather than being idle. 11 The code plugs into the Thi...

show abstract

Section: Methodsmentioning

confidence: 99%

Lightweight task offloading exploiting MPI wait times for parallel adaptive mesh refinement

Samfass

Weinzierl

Charrier

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Bamboo [24] is a source-to-source translator that translates an MPI C program into a data-driven form that overlaps communication with computation. Buettner et al [7] tried to address the issue of communication-computation overlap by extending the OpenMP runtime to include communication tasks. HCMPI (Habanero-C MPI) [9], integrates Habanero-C dynamic task-parallel programming model with the MPI message-passing interface.…”

Section: Related Workmentioning

confidence: 99%

Performance Analysis and Optimization of a Hybrid Seismic Imaging Application

Paul

Araya‐Polo

Mellor-Crummey

et al. 2016

Procedia Computer Science

View full text Add to dashboard Cite

Applications to process seismic data are computationally expensive and, therefore, employ scalable parallel systems to produce timely results. Here we describe our experiences of using performance analysis tools to gain insight into an MPI+OpenMP code developed by Shell that performs Reverse Time Migration on a cluster to produce models of the subsurface. Tuning MPI+OpenMP programs for modern platforms is difficult, and, therefore, assistance is required from performance analysis tools. These tools provided us with insights into the effectiveness of the domain decomposition strategy, the use of threaded parallelism, and functional unit utilization in individual cores. By applying insights obtained from Rice University's HPCToolkit and hardware performance counters, we were able to improve the performance of Shell's prototype distributed-memory Reverse Time Migration code by roughly 30 percent.

show abstract

“…Asynchronous communications progression in hybrid MPI+OpenMP tasks programming is discussed by David Buettner et al in [2]. They proposed an OpenMP extension to mark tasks that contain MPI communications.…”

Section: Related Workmentioning

confidence: 99%

“…Each MPI process has its own OpenMP scheduler, which executes tasks according to their precedence constraints (expressed through the depend clause) and their priorities. To address the loss of cores issue introduced by a blocking MPI communication calls, and to keep asynchronous communication progression, we propose a mix of User Level Threads (ULT), TAMPI, and MPI_Detach [2,9,16,18]. On the MPI runtime side, whenever a thread is about to block, it injects a communication progression polling function inside OpenMP, to be called on every scheduling point -as it was proposed by D. Buettner and al.…”

Section: Interoperation Between Mpi and Openmp Runtimesmentioning

confidence: 99%

“…On the MPI runtime side, whenever a thread is about to block, it injects a communication progression polling function inside OpenMP, to be called on every scheduling point -as it was proposed by D. Buettner and al. [2]. Moreover, the MPI runtime notifies the OpenMP runtime that the current task must be suspended.…”

Section: Interoperation Between Mpi and Openmp Runtimesmentioning

confidence: 99%

See 1 more Smart Citation

Communication-Aware Task Scheduling Strategy in Hybrid MPI+OpenMP Applications

Pereira

Roussel

Carribault

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

While task-based programming, such as OpenMP, is a promising solution to exploit large HPC compute nodes, it has to be mixed with data communications like MPI. However, performance or even more thread progression may depend on the underlying runtime implementations. In this paper, we focus on enhancing the application performance when an OpenMP task blocks inside MPI communications. This technique requires no additional effort on the application developers. It relies on an online task re-ordering strategy that aims at running first tasks that are sending data to other processes. We evaluate our approach on a Cholesky factorization and show that we gain around 19% of execution time on an Intel Skylake compute nodes machine -each node having two 24-core processors.

show abstract

Real Asynchronous MPI Communication in Hybrid Codes through OpenMP Communication Tasks

Cited by 7 publications

References 11 publications

Lightweight task offloading exploiting MPI wait times for parallel adaptive mesh refinement

Lightweight task offloading exploiting MPI wait times for parallel adaptive mesh refinement

Performance Analysis and Optimization of a Hybrid Seismic Imaging Application

Communication-Aware Task Scheduling Strategy in Hybrid MPI+OpenMP Applications

Contact Info

Product

Resources

About