How I learned to stop worrying about user-visible endpoints and love MPI

Abstract

MPI+threads is gaining prominence as an alternative to the traditional "MPI everywhere" model in order to better handle the disproportionate increase in the number of cores compared with other on-node resources. However, the communication performance of MPI+threads can be 100x slower than that of MPI everywhere. Both MPI users and developers are to blame for this slowdown. MPI users traditionally have not exposed logical communication parallelism. Consequently, MPI libraries have used conservative approaches, such as a global critical section, to maintain MPI's ordering constraints for MPI+threads, thus serializing access to the underlying parallel network resources and limiting performance.To enhance the communication performance of MPI+threads, researchers have proposed MPI Endpoints as a user-visible extension to the MPI-3.1 standard. MPI Endpoints allows a single process to create multiple MPI ranks within a communicator. This could, in theory, allow each thread to have a dedicated communication path to the network, thus avoiding resource contention between threads and improving performance. The onus of mapping threads to endpoints, however, would then be on domain scientists. In this paper we play the role of devil's advocate and question the need for such user-visible endpoints. We certainly agree that dedicated communication channels are critical. To what extent, however, can we hide these channels inside the MPI library without modifying the MPI standard and thus unburden the user? More important, what functionality would we lose through such abstraction? This paper answers these questions through a new implementation of the MPI-3.1 standard that uses multiple virtual communication interfaces (VCIs) inside the MPI library. VCIs abstract underlying network contexts. When users expose parallelism through existing MPI mechanisms, the MPI library maps that parallelism to the VCIs, relieving the domain scientists from worrying about endpoints. We identify cases where user-exposed parallelism on VCIs perform as well as user-visible endpoints, as well as cases where such abstraction hurts performance. CCS CONCEPTS• Software and its engineering → Massively parallel systems; Multithreading; Message oriented middleware.

MPIX Stream: An Explicit Solution to Hybrid MPI+X Programming

“…The implicit method [11,16] builds on the thesis that MPI already has a sufficient mechanism to allow users to express the inherent parallelism in their communications. Based on the outcomedictated serial execution model, any operations that do not affect MPI-specified outcomes are candidates for deserialization.…”

“…Communications from separate VCIs can be fully concurrent. For a detailed discussion of VCIs, see [16].…”

“…This performance is due to the extra critical sections introduced by MPI communications. Much research has been done on both the application side [14] and implementation side [1,11,16] to address the performance of MPI+Threads. To reach good performance, applications need to make sure that the communications can happen concurrently, and the implementations need to map the communications to multiple communication channels to allow the communication to proceed in parallel.…”

ECP Software Technology Capability Assessment Report V3.0

“…We have included support for interaction of MPICH with user-level thread (ULT) libraries [39], primarily targeting Argobots and the BOLT runtime [40]. Other issues being investigated include the design and evaluation of multiple virtual communication interfaces (VCIs) for multithreaded MPI communication [41].…”

Design of a portable implementation of partitioned point‐to‐point communication primitives

“…Unfortunately, the proposed model has foundered in the MPI Forum's standardization process. While endpoints supported thread ranks (in groups and communicators) and thus addressability in point‐to‐point and collective communication, the proposal was not adopted as improvements could be adopted in the MPI implementations transparently 5 and because it was judged that the likely impacts on transport state and receive‐queue lengths would be unacceptable with large‐count multicore nodes in current and forthcoming scalable systems 6,7 …”