Minimizing communication in sparse matrix solvers

Abstract

Data communication within the memory system of a single processor node and between multiple nodes in a system is the bottleneck in many iterative sparse matrix solvers like CG and GM-RES. Here k iterations of a conventional implementation perform k sparse-matrix-vector-multiplications and Ω(k) vector operations like dot products, resulting in communication that grows by a factor of Ω(k) in both the memory and network. By reorganizing the sparse-matrix kernel to compute a set of matrix-vector products at once and reorganizing the rest of the algorithm accordingly, we can perform k iterations by sending O(log P ) messages instead of O(k · log P ) messages on a parallel machine, and reading the matrix A from DRAM to cache just once, instead of k times on a sequential machine. This reduces communication to the minimum possible. We combine these techniques to form a new variant of GMRES. Our shared-memory implementation on an 8-core Intel Clovertown gets speedups of up to 4.3× over standard GMRES, without sacrificing convergence rate or numerical stability.

Block Iterative Methods and Recycling for Improved Scalability of Linear Solvers

“…When it comes to subspace recycling methods or block iterative methods, Gram-Schmidt schemes are often used to perform this [37]. Belos uses by default the Iterated Modified Gram-Schmidt method, but it is also possible to switch to the TSQR method, first studied in the context of CA-GMRES [47]. In our implementation, we propose to use the CholQR method [48] since its efficiency has already been proved-once again in the context of CA-GMRES [49].…”

Auto-tuning the Matrix Powers Kernel with SEJITS

“…Since an SpMV must read a matrix entry from memory for every two useful floating-point operations, making it a highly memory-bound operation, Demmel et al have proposed communicationavoiding algorithms that improve performance by trading redundant computation for memory traffic [1]. In communication-avoiding KSMs, SpMV is replaced by the matrix powers kernel, which computes Ax, A 2 x, .…”

Executing Optimized Irregular Applications Using Task Graphs within Existing Parallel Models

“…To avoid memory bottlenecks, these algorithms are modified by performance programmers to improve the algorithm's temporal and spatial locality. These optimization techniques include irregular cache blocking [1], full sparse tiling [2], and communication avoiding algorithms [3].…”