A framework for practical parallel fast matrix multiplication

Abstract

Matrix multiplication is a fundamental computation in many scientific disciplines. In this paper, we show that novel fast matrix multiplication algorithms can significantly outperform vendor implementations of the classical algorithm and Strassen's fast algorithm on modest problem sizes and shapes. Furthermore, we show that the best choice of fast algorithm depends not only on the size of the matrices but also the shape. We develop a code generation tool to automatically implement multiple sequential and shared-memory parallel variants of each fast algorithm, including our novel parallelization scheme. This allows us to rapidly benchmark over 20 fast algorithms on several problem sizes. Furthermore, we discuss a number of practical implementation issues for these algorithms on shared-memory machines that can direct further research on making fast algorithms practical.

The geometry of rank decompositions of matrix multiplication II: 3 × 3 matrices

“…We also use heuristics in order to encourage sparsity in the solutions, and a fortunate by-product of the sparsity is that the nonzero values often tend towards a discrete set of values from which an exact decomposition can be recognized. Our methods are based on techniques that have proved successful in discovering generic exact decompositions (those that have no noticeable symmetries) [1,22]; we summarize this approach in Section 3.1. Our search process can be divided into two phases.…”

“…We discuss effective choices for the regularization parameters in Section 3.3. This method works for the cases of non-square matrix multiplication and has been used to discover exact rank decompositions for many small cases [1,22], but it does not encourage solutions to reflect any symmetries.…”

“…They are such that v i,i+1 (u i+2 ) = 1, v i,i+1 (u i+3 ) = −1 (indices considered mod four). This has the advantage of v i+1,i+2 = a −i 0 v 12 where a 0 is as in (1). For v 13 there was no obvious choice of sign, and we chose v 24 = a 0 −1 (v 13 ).…”

Engineering a Delegatable and Error-Tolerant Algorithm for Counting Small Subgraphs

“…Ballard, Demmel, Holtz, Lipshitz, and Schwartz [15], Ballard, Demmel, Holtz, and Schwartz [16], and Lipshitz, Ballard, Demmel, and Schwartz [77]. Recent engineering work includes Benson and Ballard [23] and Huang, Rice, Matthews, and van de Geijn [65]. Our work differs from these works in that we seek a self-contained proof-of-concept demonstration requiring good-performance finite-field matrix multiplication on GPUs, but we do not necessarily seek the most optimized possible implementation; such optimizations are left for future work.…”

A new coding-based algorithm for finding closest pair of vectors

“…When q = 2, we use F2 and {0, 1} interchangeably throughout the paper 13. In fact, error correcting codes, as well as constructing new codes out of existing codes by concatenations to be discussed shortly, can be defined more generally over an arbitrary set of q distinct elements called alphabet of the code.…”