Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs

Abstract

Analytic, first-principles performance modeling of distributed-memory parallel codes is notoriously imprecise. Even for applications with extremely regular and homogeneous computecommunicate phases, simply adding communication time to computation time does often not yield a satisfactory prediction of parallel runtime due to deviations from the expected simple lockstep pattern caused by system noise, variations in communication time, and inherent load imbalance. In this paper, we highlight the specific cases of provoked and spontaneous desynchronization of memory-bound, bulk-synchronous pure MPI and hybrid MPI+OpenMP programs. Using simple microbenchmarks we observe that although desynchronization can introduce increased waiting time per process, it does not necessarily cause lower resource utilization but can lead to an increase in available bandwidth per core. In case of significant communication overhead, even natural noise can shove the system into a state of automatic overlap of communication and computation, improving the overall time to solution. The saturation point, i.e., the number of processes per memory domain required to achieve full memory bandwidth, is pivotal in the dynamics of this process and the emerging stable wave pattern. We also demonstrate how hybrid MPI-OpenMP programming can prevent desirable desynchronization by eliminating the bandwidth bottleneck among processes. A Chebyshev filter diagonalization application is used to demonstrate some of the observed effects in a realistic setting.

Understanding HPC Benchmark Performance on Intel Broadwell and Cascade Lake Processors

“…Desynchronization is a known phenomenon in memory-bound MPI code that can have a decisive influence on performance. See [2] for recent research.…”

Analytic Modeling of Idle Waves in Parallel Programs: Communication, Cluster Topology, and Noise Impact

“…Afzal et al [4,2,3,1] were the first to investigate the dynamics of idle waves, (de)synchronization processes, and computational wavefront formation in parallel programs with core-bound and memory-bound code, showing that nonlinear processes dominate there. Our work builds on theirs to significantly extend it for analytic modeling with further influence factors, such as communication topology, communication concurrency, system topology and noise structure.…”

“…The propagation speed of an idle wave is the speed, in ranks per second, with which it ripples through the system. Previous studies of idle wave mechanisms on silent systems [3,4] characterized the influence of execution time, communication time, communication characteristics (e.g., uni-vs. bidirectional communication patterns and eager vs. rendezvous protocols), and the number of active multi-threaded or singlethreaded MPI processes on a contended or noncontended domain. However, the scope of that work was restricted to a fixed P2P communication pattern (fourth column in Table 2 -MWSDir).…”

“…In massively parallel programs, delays can occur anytime, impeding the performance of the application. On the other hand, idle waves may also initiate desynchronization among processes, which is not necessarily disadvantageous since it can lead to automatic communication overlap [3].…”

“…The speed and overall characteristics of idle wave propagation have been the subject of some scrutiny [10,12,4,3], but a thorough analytical understanding of their dynamics with respect to the communication topology of the underlying parallel code is still lacking. There is also no investigation so far of the interaction of idle waves with global operations such as reductions, and how the system's hardware topology and the particular characteristics of system noise impact the decay of idle waves.…”