Communication optimal parallel multiplication of sparse random matrices

Abstract

Parallel algorithms for sparse matrix-matrix multiplication typically spend most of their time on inter-processor communication rather than on computation, and hardware trends predict the relative cost of communication will only increase. Thus, sparse matrix multiplication algorithms must minimize communication costs in order to scale to large processor counts.In this paper, we consider multiplying sparse matrices corresponding to Erdős-Rényi random graphs on distributed-memory parallel machines. We prove a new lower bound on the expected communication cost for a wide class of algorithms. Our analysis of existing algorithms shows that, while some are optimal for a limited range of matrix density and number of processors, none is optimal in general. We obtain two new parallel algorithms and prove that they match the expected communication cost lower bound, and hence they are optimal.

Sparse matrix multiplication: The distributed block-compressed sparse row library

“…Hence, the FLOP-to-communication ratio decreases as 1/ √ P in these weak scaling experiments. This observation can also be shown theoretically [22] The bottleneck is more pronounced on networks where the effective bandwidth decreases with increasing P . Similar observations hold for indexing overhead, which also becomes more important as the number of multiplication steps increases.…”

A Multi-Platform Evaluation of the Randomized CX Low-Rank Matrix Factorization in Spark

“…T AB: Optimizing sparse matrixmatrix multiplication is an active area of research [17], [18]; state-of-the-art implementations are bound by the memory bandwidth and heavily underutilize the compute resources.…”

Fast Matrix Multiplication with Big Sparse Data

“…Parallelisation and indexing techniques for sparse matrices multiplication were implemented by B u l u c and G i l b e r t [7]. The communication overhead problem of sparse matrices multiplication was solved by B a l l a r d et al [8]. The parallelisation technique for sparse tensor matrix multiplication was proposed by S m i t h et al [9].…”

“…The parallelisation technique for sparse tensor matrix multiplication was proposed by S m i t h et al [9]. The above approaches [7][8][9] are not suitable for Big Data applications. Proper care should be taken by the programmer regarding the data distribution, replication, load balancing, communication overhead etc.…”