Improving the performance of MPI derived datatypes by optimizing memory-access cost

Abstract

The IntroductionThe MPI (Message Passing Interface) Standard is widely used in parallel computing for writing distributedmemory parallel programs [1,2]. MPI has a number of features that provide both convenience and high performance. One of the important features is the concept of derived datatypes. Derived datatypes enable users to describe noncontiguous memory layouts compactly and to use this compact representation in MPI communication functions. Derived datatypes also enable an MPI implementation to optimize the transfer of noncontiguous data. For example, if the underlying communication mechanism supports noncontiguous data transfers, the MPI implementation can communicate the data directly without packing it into a contiguous buffer. On the other hand, if packing into a contiguous buffer is necessary, the MPI implementation can pack the data and send it contiguously. In practice, however, many MPI implementations perform poorly with derived datatypes-to the extent that users often resort to packing the data manually into a contiguous buffer and then calling MPI. Such usage clearly defeats the purpose of having derived datatypes in the MPI Standard. Since noncontiguous communication occurs commonly in many applications (for example, fast Fourier transform, array redistribution, and finite-element codes), improving the performance of derived datatypes has significant value.The performance of derived datatypes can be improved in two ways. One way is to improve the data structures used to store derived datatypes internally in the MPI implementation, so that, in an MPI communication call, the implementation can quickly decode the information represented by the datatype. Research has already been done in this area, mainly in using data structures that allow a stack-based approach to parsing a datatype, rather than making recursive function calls, which are expensive [3,4]. Another area for improvement is to use optimized algorithms for packing noncontiguous data into a contiguous buffer in a way that the user could not do easily without advanced knowledge of the memory architecture. This latter area is the focus of this paper. To our knowledge, no other MPI implementations use memory-optimization techniques for packing noncontiguous data in their derived-datatype code (for example, see the results with IBM's MPI in Figure 8).Interprocess communication can be considered as a combination of memory communication and network communication as defined in [5]. Memory communication (or memory copying) is the transfer of data from the user's buffer to the local network buffer (or shared-memory buffer) and vice versa. Network communication is the movement of data between source

MPI datatype processing using runtime compilation

“…Furthermore, Byna et al show that tuning the datatype interpreter for the memory-hierarchy on the target machine (cache and page sizes, etc.) can lead to performance improvements [3]. They do this by providing different packing functions which are parametrized with information about the actual datatype as well as information about the memory hierarchy of the node.…”

Micro-applications for Communication Data Access Patterns and MPI Datatypes

“…Previous work in the area of MPI derived datatypes focuses on improving its performance, either by improving the way derived datatypes are represented in MPI or by using more cache efficient strategies for packing and unpacking the datatype to and from a contiguous buffer [6]. Interconnect features such as RDMA Scatter/Gather operations [20] have also been considered.…”

“…Not many scientific codes leverage MPI DDTs, even though their usage would be appropriate in many cases. One of the reasons might be that current MPI implementations in some cases still fail to deliver the expected performance, as shown by Gropp et al in [9], even though a lot of work is done on improving DDT implementations [6,18,20]. Most of this work is guided by a small number of micro-benchmarks.…”

Runtime optimization of vector operations on large scale SMP clusters

“…To address the needs of these applications, communication layers provide higher level primitives for such transfers. These operations are usually referred to in literature as vector or strided and they have been shown to improve application performance [19,23,7]. Most of the native communication layers, such as Elan, InfiniBand Verbs, IBM LAPI provide an API for these operations.…”