Fast matrix multiplication is stable

Demmel, James; Dumitriu, Ioana; Holtz, Olga; Kleinberg, Robert

doi:10.1007/s00211-007-0061-6

Cited by 60 publications

(67 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The practical implementation of Pan's algorithm is presented by Kaporini [23,24]. New approaches are emerging recently, which promise to be practical and numerically stable [7,12]. The fastest to date is by Coppersmith and Winograd [8] O(n 2.376 ).…”

Section: Related Workmentioning

confidence: 99%

Adaptive Strassen's matrix multiplication

D'Alberto

Nicolau

2007

Proceedings of the 21st Annual International Conference on Supercomputing

View full text Add to dashboard Cite

Strassen's matrix multiplication (MM) has benefits with respect to any (highly tuned) implementations of MM because Strassen's reduces the total number of operations. Strassen achieved this operation reduction by replacing computationally expensive MMs with matrix additions (MAs). For architectures with simple memory hierarchies, having fewer operations directly translates into an efficient utilization of the CPU and, thus, faster execution. However, for modern architectures with complex memory hierarchies, the operations introduced by the MAs have a limited in-cache data reuse and thus poor memory-hierarchy utilization, thereby overshadowing the (improved) CPU utilization, and making Strassen's algorithm (largely) useless on its own.In this paper, we investigate the interaction between Strassen's effective performance and the memory-hierarchy organization. We show how to exploit Strassen's full potential across different architectures. We present an easy-to-use adaptive algorithm that combines a novel implementation of Strassen's idea with the MM from automatically tuned linear algebra software (ATLAS) or GotoBLAS. An additional advantage of our algorithm is that it applies to any size and shape matrices and works equally well with row or column major layout. Our implementation consists of introducing a final step in the ATLAS/GotoBLAS-installation process that estimates whether or not we can achieve any additional speedup using our Strassen's adaptation algorithm. Then we install our codes, validate our estimates, and determine the specific performance.We show that, by the right combination of Strassen's with AT-LAS/GotoBLAS, our approach achieves up to 30%/22% speed-up versus ATLAS/GotoBLAS alone on modern high-performance single processors. We consider and present the complexity and the numerical analysis of our algorithm, and, finally, we show performance for 17 (uniprocessor) systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Adaptive Strassen's matrix multiplication

D'Alberto

Nicolau

2007

Proceedings of the 21st Annual International Conference on Supercomputing

View full text Add to dashboard Cite

show abstract

“…23]) and extended by Bini and Lotti [6] to a larger class of algorithms. In prior work [23] we showed that such a bound holds for a new class of fast algorithms depending on group-theoretic methods [18] and [17], which include an algorithm that runs asymptotically as fast as the fastest known method due to Coppersmith and Winograd [19], which runs in about O(n 2.38 ) operations. Using a result of Raz [43], that work also showed that any fast matrix multiplication algorithm running in O(n ω+η ) arithmetic operations can be converted to one that satisfies (1) and also runs in O(n ω+η ) arithmetic operations.…”

Section: Introductionmentioning

confidence: 99%

“…In prior results [23] we showed that any fast matrix multiplication algorithm running in time O(n ω+η ) was either stable or could be converted into a stable algorithm that also ran in O(n ω+η ) operations. Combined with the results in this paper, this lets us state that all linear algebra operations can also be done stably in O(n ω+η ) operations.…”

Section: Introductionmentioning

confidence: 99%

“…

AbstractIn [23] we showed that a large class of fast recursive matrix multiplication algorithms is stable in a normwise sense, and that in fact if multiplication of n-by-n matrices can be done by any algorithm in O(n ω+η ) operations for any η > 0, then it can be done stably in O(n ω+η ) operations for any η > 0. Here we extend this result to show that essentially all standard linear algebra operations, including LU decomposition, QR decomposition, linear equation solving, matrix inversion, solving least squares problems, (generalized) eigenvalue problems and the singular value decomposition can also be done stably (in a normwise sense) in O(n ω+η ) operations.

…”

mentioning

confidence: 99%

See 1 more Smart Citation

Fast linear algebra is stable

2007

Self Cite

View full text Add to dashboard Cite

In [23] we showed that a large class of fast recursive matrix multiplication algorithms is stable in a normwise sense, and that in fact if multiplication of n-by-n matrices can be done by any algorithm in O(n ω+η ) operations for any η > 0, then it can be done stably in O(n ω+η ) operations for any η > 0. Here we extend this result to show that essentially all standard linear algebra operations, including LU decomposition, QR decomposition, linear equation solving, matrix inversion, solving least squares problems, (generalized) eigenvalue problems and the singular value decomposition can also be done stably (in a normwise sense) in O(n ω+η ) operations.

show abstract

“…These algorithms are numerically stable [Demmel et al 2006] because they are based on the Discrete Fourier Transform (DFT) kernel computation. However, there have not been any experimental quantification of the benefits of such approaches.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Winograd's matrix multiplications

D'Alberto

Nicolau

2009

ACM Trans. Math. Softw.

View full text Add to dashboard Cite

Modern architectures have complex memory hierarchies and increasing parallelism (e.g., multicores). These features make achieving and maintaining good performance across rapidly changing architectures increasingly difficult. Performance has become a complex trade-off, not just a simple matter of counting cost of simple CPU operations.We present a novel, hybrid, and adaptive recursive Strassen-Winograd's matrix multiplication (MM) that uses automatically tuned linear algebra software (ATLAS) or GotoBLAS. Our algorithm applies to any size and shape matrices stored in either row or column major layout (in double-precision in this work) and thus is efficiently applicable to both C and FORTRAN implementations. In addition, our algorithm divides the computation into equivalent in-complexity sub-MMs and does not require any extra computation to combine the intermediary sub-MM results.We achieve up to 22% execution-time reduction versus GotoBLAS/ATLAS alone for a single core system and up to 19% for a 2 dual-core processor system. Most importantly, even for small matrices such as 1500×1500, our approach attains already 10% execution-time reduction and, for MM of matrices larger than 3000×3000, it delivers performance that would correspond, for a classic O(n 3 ) algorithm, to faster-than-processor peak performance (i.e., our algorithm delivers the equivalent of 5 GFLOPS performance on a system with 4.4 GFLOPS peak performance and where GotoBLAS achieves only 4 GFLOPS). This is a result of the savings in operations (and thus FLOPS). Therefore, our algorithm is faster than any classic MM algorithms could ever be for matrices of this size. Furthermore, we present experimental evidence based on established methodologies found in the literature that our algorithm is, for a family of matrices, as accurate as the classic algorithms.

show abstract

Fast matrix multiplication is stable

Cited by 60 publications

References 20 publications

Adaptive Strassen's matrix multiplication

Adaptive Strassen's matrix multiplication

Fast linear algebra is stable

Adaptive Winograd's matrix multiplications

Contact Info

Product

Resources

About