Trading Replication for Communication in Parallel Distributed-Memory Dense Solvers

Irony, Dror; Toledo, Sivan

doi:10.1142/s0129626402000847

Cited by 10 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar asymptotic optimality results are proven for the 3D matrix multiplication [7] and 2.5D matrix multiplication and LU factorization [15]. An early implementation of fully 3D distributed algorithms for triangular solve and LU factorization without pivoting is proposed in [16] that are asymptoticaly optimal for the total volume of communications. The 2.5D algorithms bring a continuum between 2D and 3D algorithms where the trade-off between memory footprint and communication is controlled by a parameter.…”

Section: Related Workmentioning

confidence: 52%

Symmetric Block-Cyclic Distribution: Fewer Communications Leads to Faster Dense Cholesky Factorization

Beaumont

Duchon

Eyraud-Dubois

et al. 2022

SC22: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

We consider the distributed Cholesky factorization on homogeneous nodes. Inspired by recent progress on asymptotic lower bounds on the total communication volume required to perform Cholesky factorization, we present an original data distribution, Symmetric Block Cyclic (SBC), designed to take advantage of the symmetry of the matrix. We prove that SBC reduces the overall communication volume between nodes by a factor of square root of 2 compared to the standard 2D blockcyclic distribution. SBC can easily be implemented within the paradigm of task-based runtime systems. Experiments using the Chameleon library over the StarPU runtime system demonstrate that the SBC distribution reduces the communication volume as expected, and also achieves better performance and scalability than the classical 2D block-cyclic allocation scheme in all configurations. We also propose a 2.5D variant of SBC and prove that it further improves the communication and performance benefits.

show abstract

Section: Related Workmentioning

confidence: 52%

Symmetric Block-Cyclic Distribution: Fewer Communications Leads to Faster Dense Cholesky Factorization

Beaumont

Duchon

Eyraud-Dubois

et al. 2022

SC22: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…Parallel TRSM algorithms with 3D processor grids can reduce the communication cost in an analogous fashion to matrix multiplication. Irony and Toledo [20] presented the first parallelization of the recursive TRSM algorithm with a 3D processor grid. They demonstrated that the communication volume of their parallelization is O(nkp 1/3 + n 2 p 1/3 ).…”

Section: ) Recursive Triangular Matrix Solve For Multiple Rightmentioning

confidence: 99%

Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations

Wicky¹,

Solomonik

Hoefler³

2017

2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

We present a new parallel algorithm for solving triangular systems with multiple right hand sides (TRSM). TRSM is used extensively in numerical linear algebra computations, both to solve triangular linear systems of equations as well as to compute factorizations with triangular matrices, such as Cholesky, LU, and QR. Our algorithm achieves better theoretical scalability than known alternatives, while maintaining numerical stability, via selective use of triangular matrix inversion. We leverage the fact that triangular inversion and matrix multiplication are more parallelizable than the standard TRSM algorithm. By only inverting triangular blocks along the diagonal of the initial matrix, we generalize the usual way of TRSM computation and the full matrix inversion approach. This flexibility leads to an efficient algorithm for any ratio of the number of right hand sides to the triangular matrix dimension. We provide a detailed communication cost analysis for our algorithm as well as for the recursive triangular matrix inversion. This cost analysis makes it possible to determine optimal block sizes and processor grids a priori. Relative to the best known algorithms for TRSM, our approach can require asymptotically fewer messages, while performing optimal amounts of computation and communication in terms of words sent.

show abstract

“…A first challenge to 2DBC was brought by the development of 3D and 2.5D algorithms [1], [2]. These algorithms are based on a 3D representation of dense linear algebra algorithms whose complexity is in O(N 3 ) where N is the size of the matrix (products, dense factorizations).…”

Section: Introductionmentioning

confidence: 99%

Data Distribution Schemes for Dense Linear Algebra Factorizations on Any Number of Nodes

Beaumont

Collin

Eyraud-Dubois

et al. 2023

2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

In this paper, we consider the problem of distributing the tiles of a dense matrix onto a set of homogeneous nodes. We consider both the case of non-symmetric (LU) and symmetric (Cholesky) factorizations. The efficiency of the well-known 2D Block-Cyclic (2DBC) distribution degrades significantly if the number of nodes P cannot be written as the product of two close numbers. Similarly, the recently introduced Symmetric Block Cyclic (SBC) distribution is only valid for specific values of P . In both contexts, we propose generalizations of these distributions to adapt them to any number of nodes. We show that this provides improvements to existing schemes (2DBC and SBC) both in theory and in practice, using the flexibility and ease of programming induced by task-based runtime systems like Chameleon and StarPU.

show abstract

Trading Replication for Communication in Parallel Distributed-Memory Dense Solvers

Cited by 10 publications

References 10 publications

Symmetric Block-Cyclic Distribution: Fewer Communications Leads to Faster Dense Cholesky Factorization

Symmetric Block-Cyclic Distribution: Fewer Communications Leads to Faster Dense Cholesky Factorization

Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations

Data Distribution Schemes for Dense Linear Algebra Factorizations on Any Number of Nodes

Contact Info

Product

Resources

About