Exploiting parallelism in matrix-computation kernels for symmetric multiprocessor systems

Abstract

We present a simple and efficient methodology for the development, tuning, and installation of matrix algorithms such as the hybrid Strassen's and Winograd's fast matrix multiply or their combination with the 3M algorithm for complex matrices (i.e., hybrid: a recursive algorithm as Strassen's until a highly tuned BLAS matrix multiplication allows performance advantages). We investigate how modern Symmetric Multiprocessor (SMP) architectures present old and new challenges that can be addressed by the combination of an algorithm design with careful and natural parallelism exploitation at the function level (optimizations) such as function-call parallelism, function percolation, and function software pipelining. We have three contributions: first, we present a performance overview for double- and double-complex-precision matrices for state-of-the-art SMP systems; second, we introduce new algorithm implementations: a variant of the 3M algorithm and two new different schedules of Winograd's matrix multiplication (achieving up to 20% speedup with respect to regular matrix multiplication). About the latter Winograd's algorithms: one is designed to minimize the number of matrix additions and the other to minimize the computation latency of matrix additions; third, we apply software pipelining and threads allocation to all the algorithms and we show how this yields up to 10% further performance improvements.

Recursion based parallelization of exact dense linear algebra routines for Gaussian elimination

“…In particular, numerical linear algebra based on Strassen's algorithm (if numerical stability issues have been considered acceptable) should clearly benefit from most of its results. Related work on the parallelization of the sub-cubic numerical linear algebra include [1,24,6,25,2].…”

A framework for practical parallel fast matrix multiplication

“…There are several sequential implementations of Strassen's fast matrix multiplication algorithm [2,11,17], and parallel versions have been implemented for both shared-memory [9,25] and distributedmemory architectures [3,13]. For our parallel algorithms in Section 4, we use the ideas of breadth-first and depth-first traversals of the recursion trees, which were first considered by Kumar et al [25] and Ballard et al [3] for minimizing memory footprint and communication.…”

“…However, this metric lets us compare all of the algorithms on an inverse-time scale, normalized by problem size [13,27]. We compare our code-generated Strassen implementation with MKL's dgemm and a tuned implementation of Strassen-Winograd from D'Alberto et al [9] (recall that Strassen-Winograd performs the same number of multiplications but fewer matrix additions than Strassen's algorithm). The code generator's implementation outperforms MKL and is competitive with the tuned implementation.…”

“…The code generator's implementation outperforms MKL and is competitive with the tuned implementation. (3)) of our code generator's implementation of Strassen's algorithm against MKL's dgemm and a tuned implementation of the Strassen-Winograd algorithm [9]. The problems sizes are square.…”