Fast linear algebra is stable

Demmel, James; Dumitriu, Ioana; Holtz, Olga

doi:10.1007/s00211-007-0114-x

Cited by 169 publications

(200 citation statements)

References 46 publications

Supporting

Mentioning

200

Contrasting

Order By: Relevance

“…Indeed, "block" algorithms relying on matrix multiplication are used in practice for many linear algebra operations [1,3], and have been shown to be stable assuming only the error bound (4) [12]. In a companion paper [11], we show that while stable these earlier block algorithms are not asymptotically as fast as matrix multiplication. However, [11] also shows there are variants of these block algorithms for operations like QR decomposition, linear equation solving and determinant computation that are both stable and as fast as matrix multiplication.…”

Section: Stability Of Linear Algebra Algorithms Based On Matrix Multimentioning

confidence: 91%

Fast matrix multiplication is stable

et al. 2007

Self Cite

View full text Add to dashboard Cite

We perform forward error analysis for a large class of recursive matrix multiplication algorithms in the spirit of [D. Bini and G. Lotti, Stability of fast algorithms for matrix multiplication, Numer. Math. 36 (1980), 63-72]. As a consequence of our analysis, we show that the exponent of matrix multiplication (the optimal running time) can be achieved by numerically stable algorithms. We also show that new group-theoretic algorithms proposed in [H. Cohn, and C. Umans, A group-theoretic approach to fast matrix multiplication, FOCS 2003, 438-449] and [H. Cohn, R. Kleinberg, B. Szegedy and C. Umans, Group-theoretic algorithms for matrix multiplication, FOCS 2005, 379-388] are all included in the class of algorithms to which our analysis applies, and are therefore numerically stable. We perform detailed error analysis for three specific fast group-theoretic algorithms.

show abstract

Section: Stability Of Linear Algebra Algorithms Based On Matrix Multimentioning

confidence: 91%

Fast matrix multiplication is stable

et al. 2007

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the symmetric eigenproblem and SVD, there are such algorithms that begin by reduction to a condensed form. But for the nonsymmetric eigenproblem, the only known algorithm attaining the expected lower bound does not initially reduce to condensed form, and is not based on QR iteration [DDH07,BDD11].…”

Section: Eigenvalue and Singular Value Problemsmentioning

confidence: 99%

Minimizing Communication in Numerical Linear Algebra

Ballard¹,

Demmel²,

Holtz³

et al. 2011

SIAM J. Matrix Anal. & Appl.

Self Cite

215

308

View full text Add to dashboard Cite

Abstract. In 1981 Hong and Kung proved a lower bound on the amount of communication (amount of data moved between a small, fast memory and large, slow memory) needed to perform dense, n-by-n matrix-multiplication using the conventional O(n 3 ) algorithm, where the input matrices were too large to fit in the small, fast memory. In 2004 Irony, Toledo and Tiskin gave a new proof of this result and extended it to the parallel case (where communication means the amount of data moved between processors). In both cases the lower bound may be expressed as Ω(#arithmetic operations / √ M ), where M is the size of the fast memory (or local memory in the parallel case). Here we generalize these results to a much wider variety of algorithms, including LU factorization, Cholesky factorization, LDL T factorization, QR factorization, Gram-Schmidt algorithm, algorithms for eigenvalues and singular values, i.e., essentially all direct methods of linear algebra.The proof works for dense or sparse matrices, and for sequential or parallel algorithms. In addition to lower bounds on the amount of data moved (bandwidth-cost), we get lower bounds on the number of messages required to move it (latency-cost).We extend our lower bound technique to compositions of linear algebra operations (like computing powers of a matrix), to decide whether it is enough to call a sequence of simpler optimal algorithms (like matrix multiplication) to minimize communication, or if we can do better. We give examples of both. We also show how to extend our lower bounds to certain graph theoretic problems.We point out recently designed algorithms that attain many of these lower bounds.

show abstract

“…Thus, using Cholesky-QR we plan to formulate many other numerical linear algebra operations with minimal communication. As an alternative, we are also looking into adjusting the algorithms for computing QR, eigenvalue decompositions, and the SVD which use Strassen's algorithm [8] to using our 2.5D matrix multiplication algorithm instead. Further, we plan to look for the most efficient and stable 2.5D QR factorization algorithms.…”

Section: Future Workmentioning

confidence: 99%

Communication-Optimal Parallel 2.5D Matrix Multiplication and LU Factorization Algorithms

Solomonik

Demmel

2011

Lecture Notes in Computer Science

Self Cite

195

239

View full text Add to dashboard Cite

Abstract. One can use extra memory to parallelize matrix multiplication by storing p 1/3 redundant copies of the input matrices on p processors in order to do asymptotically less communication than Cannon's algorithm [2], and be faster in practice [1]. We call this algorithm "3D" because it arranges the p processors in a 3D array, and Cannon's algorithm "2D" because it stores a single copy of the matrices on a 2D array of processors. We generalize these 2D and 3D algorithms by introducing a new class of "2.5D algorithms". For matrix multiplication, we can take advantage of any amount of extra memory to store c copies of the data, for any c ∈ {1, 2, ..., p 1/3 }, to reduce the bandwidth cost of Cannon's algorithm by a factor of c 1/2 and the latency cost by a factor c 3/2 . We also show that these costs reach the lower bounds [13,3], modulo polylog(p) factors. We similarly generalize LU decomposition to 2.5D and 3D, including communication-avoiding pivoting, a stable alternative to partial-pivoting [7]. We prove a novel lower bound on the latency cost of 2.5D and 3D LU factorization, showing that while c copies of the data can also reduce the bandwidth by a factor of c 1/2 , the latency must increase by a factor of c 1/2 , so that the 2D LU algorithm (c = 1) in fact minimizes latency. Preliminary results of 2.5D matrix multiplication on a Cray XT4 machine also demonstrate a performance gain of up to 3X with respect to Cannon's algorithm. Careful choice of c also yields up to a 2.4X speed-up over 3D matrix multiplication, due to a better balance between communication costs.

show abstract

Fast linear algebra is stable

Cited by 169 publications

References 46 publications

Fast matrix multiplication is stable

Fast matrix multiplication is stable

Minimizing Communication in Numerical Linear Algebra

Communication-Optimal Parallel 2.5D Matrix Multiplication and LU Factorization Algorithms

Contact Info

Product

Resources

About