Improving all-reduce collective operations for imbalanced process arrival patterns

Abstract

Two new algorithms for the all-reduce operation, optimized for imbalanced process arrival patterns (PAPs) are presented: (i) sorted linear tree (SLT), (ii) pre-reduced ring (PRR) as well as a new way of on-line PAP detection, including process arrival time (PAT) estimations and their distribution between cooperating processes was introduced. The idea, pseudo-code, implementation details, benchmark for performance evaluation and a real case example for machine learning are provided. The results of the experiments were described and analyzed, showing that the proposed solution has high scalability and improved performance in comparison with the usually used ring and Rabenseifner algorithms.Collective communication [2] is frequently used by the programmers and designers of parallel programs, especially in high performance computing (HPC) applications related to scientific simulations and data analysis, including machine learning calculations. Usually, collective operations, e.g. implemented in MPI [6], are based on algorithms optimized for the simultaneous entering of all participants into the operation, i.e. they do not take into consideration possible differences in process arrival times (PATs), thus, in real environment, where such imbalances are ubiquitous, they can have significant performance

Process arrival pattern aware algorithms for acceleration of scatter and gather operations

“…In [23] we proposed two new, hardware agnostic, allreduce algorithms optimized for imbalanced PAP occurrence, the solution included a PAP detection mechanism based on progress monitoring by an additional background thread placed in every process participating in the collective operation. A benchmark evaluating the performance of the algorithms was described and experimental results comparing with other typically used algorithms were provided.…”

“…The implementation uses C language (v. C99, compiled by GCC v. 7.3.0 with -O 3 optimization), with OpenMPI [10] (v. 3.0.0) for processes/nodes message exchange, POSIX Threads [4] (v. 2.12) for intranode communication and synchronization, and GLibc (v. 2.0) for dynamic data structures' management. The similar approach was used in [23].…”

“…The proposed model is an extension of a model described in [23], which considers parallel processing in a homogeneous compute cluster environment and focuses on the process arrival and exit patterns. The aforementioned work is extended by a definition of operation run time, which is helpful in evaluation of non-symmetric collective algorithms, i.e.…”

“…In [23] we proposed an iterative model of computations along with an additional, background thread for monitoring purposes. The thread performs the data exchange during the computation phase (when the network is usually underused), and provides the information about the progress of the computations to all cooperating processes.…”

Improving Clairvoyant: reduction algorithm resilient to imbalanced process arrival patterns

“…In our case we optimize collective operations by providing some algorithms (partially) resilient to imbalanced PAP environment, e.g. [15,16].…”