Fine-Grained Multithreading Support for Hybrid Threaded MPI Programming

Abstract

As high-end computing systems continue to grow in scale, recent advances in multi- and many-core architectures have pushed such growth toward more dense architectures, that is, more processing elements per physical node, rather than more physical nodes themselves. Although a large number of scientific applications have relied so far on an MPI-everywhere model for programming high-end parallel systems; this model may not be sufficient for future machines, given their physical constraints such as decreasing amounts of memory per processing element and shared caches. As a result, application and computer scientists are exploring alternative programming models that involve using MPI between address spaces and some other threaded model, such as OpenMP, Pthreads, or Intel TBB, within an address space. Such hybrid models require efficient support from an MPI implementation for MPI messages sent from multiple threads simultaneously. In this paper, we explore the issues involved in designing such an implementation. We present four approaches to building a fully thread-safe MPI implementation, with decreasing levels of critical-section granularity (from coarse-grain locks to fine-grain locks to lock-free operations) and correspondingly increasing levels of complexity. We present performance results that demonstrate the performance implications of the different approaches.

Final Report for Enhancing the MPI Programming Model for PetaScale Systems

“…Work early in the project was performed in collaboration with both Argonne and the IBM Blue Gene team [10]. Work on fine grain multithreading support showed how to avoid excessive lock overhead in an MPI implementation [3,2]. Recent work included a new algorithm for efficient allocation of context ids in MPI fixes a subtle race condition in the algorithm that had been used in MPICH; this new algorithm retains the efficient behavior for the expected case [9].…”

Minimizing MPI Resource Contention in Multithreaded Multicore Environments

“…In addition to this common object allocation layer used for all MPI objects, MPICH2 provides another small optimization above this layer for MPI_Request objects [3]. In general, an MPI implementation must allocate a request object for each communication operation such as MPI_Send or MPI_Recv.…”

“…For example, keeping track of |T | by atomically incrementing or decrementing a shared counter on every modification of T would incur a severe performance penalty. As mentioned in Section III, MPICH2 currently uses a thread-local storage optimization [3] to manage request allocation. This optimization eliminates virtually all contention from request allocation.…”

“…To this end we present a minor variation on our threaded message rate benchmark [3] that sends and receives messages bidirectionally in order to provide the most parallel and efficient baseline possible. This benchmark measures the aggregate message rate for N threads in a single MPI process, each sending to and receiving from a corresponding peer process on a separate node.…”

Implementing MPI on Windows: Comparison with Common Approaches on Unix

“…When a lock is released, a thread waiting for the global lock gets access to the lock and performs progress on its MPI communication. We are also developing a more efficient version of MPICH2 that supports finer-grained locks [1].…”