Communication-Optimal Parallel Recursive Rectangular Matrix Multiplication

Demmel, James; Eliahu, David; Fox, Armando; Kamil, Shoaib; Lipshitz, Benjamin; Schwartz, Oded; Spillinger, Omer

doi:10.1109/ipdps.2013.80

Cited by 100 publications

(103 citation statements)

References 29 publications

Supporting

Mentioning

101

Contrasting

Order By: Relevance

“…We also present a new 3D recursive algorithm which is a parallelization of a sequential recursive algorithm using the techniques of [3,13]. Although we have assumed that the input matrices are square, the recursive algorithm will use rectangular matrices for subproblems.…”

Section: D Recursive Algorithmmentioning

confidence: 99%

“…We obtain two new communication-optimal algorithms. Our 3D iterative and recursive algorithms (see Sections 4.3 and 4.4) are adaptations of dense ones [13,25], though an important distinction is that the sparse algorithms do not require extra local memory to minimize communication. We also optimize an existing algorithm, Sparse SUMMA, to be communication-optimal in some cases.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Communication optimal parallel multiplication of sparse random matrices

Ballard

Buluç

Demmel

et al. 2013

Proceedings of the Twenty-Fifth Annual ACM Symposium on Parallelism in Algorithms and Architectures

Self Cite

View full text Add to dashboard Cite

Parallel algorithms for sparse matrix-matrix multiplication typically spend most of their time on inter-processor communication rather than on computation, and hardware trends predict the relative cost of communication will only increase. Thus, sparse matrix multiplication algorithms must minimize communication costs in order to scale to large processor counts.In this paper, we consider multiplying sparse matrices corresponding to Erdős-Rényi random graphs on distributed-memory parallel machines. We prove a new lower bound on the expected communication cost for a wide class of algorithms. Our analysis of existing algorithms shows that, while some are optimal for a limited range of matrix density and number of processors, none is optimal in general. We obtain two new parallel algorithms and prove that they match the expected communication cost lower bound, and hence they are optimal.

show abstract

Section: D Recursive Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Communication optimal parallel multiplication of sparse random matrices

Ballard

Buluç

Demmel

et al. 2013

Proceedings of the Twenty-Fifth Annual ACM Symposium on Parallelism in Algorithms and Architectures

Self Cite

View full text Add to dashboard Cite

show abstract

“…In a memory constrained case (2D), each processor needs to hold n 2 / √ p portion of the data. In the communication optimal case, each processor should have enough memory to hold a larger partition of the input matrices (A and B) estimated by n 2 /p 2/3 words [26]. Having such portions of data could involve at most n 2 /p 2/3…”

Section: Analytical Estimate Of the Data Movement Of The Proxy Benchmarkmentioning

confidence: 99%

Cross-scale efficient tensor contractions for coupled cluster computations through multiple programming model backends

Ibrahim

Epifanovsky

Williams

et al. 2017

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

Coupled-cluster methods provide highly accurate models of molecular structure through explicit numerical calculation of tensors representing the correlation between electrons. These calculations are dominated by a sequence of tensor contractions, motivating the development of numerical libraries for such operations. While based on matrix-matrix multiplication, these libraries are specialized to exploit symmetries in the molecular structure and in electronic interactions, and thus reduce the size of the tensor representation and the complexity of contractions. The resulting algorithms are irregular and their parallelization has been previously achieved via the use of dynamic scheduling or specialized data decompositions. We introduce our efforts to extend the Libtensor framework to work in the distributed memory environment in a scalable and energy-efficient manner. We achieve up to 240× speedup compared with the optimized shared memory implementation of Libtensor. We attain scalability to hundreds of thousands of compute cores on three distributedmemory architectures, (Cray XC30 and XC40, and IBM Blue Gene/Q), and on a heterogeneous GPU-CPU system (Cray XK7). As the bottlenecks shift from being compute-bound DGEMM's to communication-bound collectives as the size of the molecular system scales, we adopt two radically different parallelization approaches for handling load-imbalance, tasking and bulk synchronous models. Nevertheless, we preserve a unified interface to both programming models to maintain the productivity of computational quantum chemists.

show abstract

“…In 2012, Ballard [10] proposed a communication-optimal Strassen's algorithm which performs better than any previous classical or Strassenbased parallel algorithm for square matrix multiplication. Later, Demmel [39] proposed a communication-optimal recursive algorithm for rectangular matrix multiplication, applying communication-optimal theory to all matrix dimensions.…”

Section: Related Workmentioning

confidence: 99%

A Cloud-Friendly Communication-Optimal Implementation for Strassen's Matrix Multiplication Algorithm

Zhou

2015

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYDue to its on-demand and pay-as-you-go properties, cloud computing has become an attractive alternative for HPC applications. However, communication-intensive applications with complex communication patterns still cannot be performed efficiently on cloud platforms, which are equipped with MapReduce technologies, such as Hadoop and Spark. In particular, one major obstacle is that MapReduce's simple programming model cannot explicitly manipulate data transfers between compute nodes. Another obstacle is cloud's relatively poor network performance compared with traditional HPC platforms. The traditional Strassen's algorithm of square matrix multiplication has a recursive and complex pattern on the HPC platform. Therefore, it cannot be directly applied to the cloud platform. In this paper, we demonstrate how to make Strassen's algorithm with complex communication patterns "cloud-friendly". By reorganizing Strassen's algorithm in an iterative pattern, we completely separate its computations and communications, making it fit to MapReduce programming model. By adopting a novel data/task parallel strategy, we solve Strassen's data dependency problems, making it well balanced. This is the first instance of Strassen's algorithm in MapReduce-style systems, which also matches Strassen's communication lower bound. Further experimental results show that it achieves a speedup ranging from 1.03× to 2.50× over the classical Θ(n 3 ) algorithm. We believe the principle can be applied to many other complex scientific applications.

show abstract

Communication-Optimal Parallel Recursive Rectangular Matrix Multiplication

Cited by 100 publications

References 29 publications

Communication optimal parallel multiplication of sparse random matrices

Communication optimal parallel multiplication of sparse random matrices

Cross-scale efficient tensor contractions for coupled cluster computations through multiple programming model backends

A Cloud-Friendly Communication-Optimal Implementation for Strassen's Matrix Multiplication Algorithm

Contact Info

Product

Resources

About