Symmetric Block-Cyclic Distribution: Fewer Communications Leads to Faster Dense Cholesky Factorization

Abstract

We consider the distributed Cholesky factorization on homogeneous nodes. Inspired by recent progress on asymptotic lower bounds on the total communication volume required to perform Cholesky factorization, we present an original data distribution, Symmetric Block Cyclic (SBC), designed to take advantage of the symmetry of the matrix. We prove that SBC reduces the overall communication volume between nodes by a factor of square root of 2 compared to the standard 2D blockcyclic distribution. SBC can easily be implemented within the paradigm of task-based runtime systems. Experiments using the Chameleon library over the StarPU runtime system demonstrate that the SBC distribution reduces the communication volume as expected, and also achieves better performance and scalability than the classical 2D block-cyclic allocation scheme in all configurations. We also propose a 2.5D variant of SBC and prove that it further improves the communication and performance benefits.

Data Distribution Schemes for Dense Linear Algebra Factorizations on Any Number of Nodes

“…Sequential algorithms matching those bound are also presented. In the parallel setting, an improved distribution based on the same ideas called SBC is developed in [3] which take advantage of the symmetry of the input to reduces the communications. This cyclic distribution ensures a good load balancing and is shown to outperform the 2DBC distribution for Cholesky factorization and SYRK operation.…”

“…A second challenge to 2DBC has recently been brought in the case of symmetric problems, such as the multiplication of symmetric matrices (SYRK) or the Cholesky factorization (see Section II-A). In this context, it has been shown that it is possible to design specialized data distribution schemes such as SBC [3] (Symmetric Block Cyclic distribution) which generate strictly less communication than 2DBC, even among 2D data distributions. These communication schemes are typically based on a cyclic distribution, but with a larger pattern in which each node can appear several times.…”

“…From this information, communications are automatically managed by the runtime, with general purpose strategies to overlap as much as possible communications with computations. Thus, it has been demonstrated [3] that significant increased performance can be obtained when using complex data allocation patterns, compared to those obtained with 2DBC, without significant effort for the implementation. Moreover, the use of the runtime allows to avoid synchronizations between the different steps of a LU or Cholesky factorization, and thus allows to obtain much better performance than MPI-based codes that are usually more synchronized due to implementation issues.…”

“…We then consider the case of distributions tailored to symmetric problems. The SBC distribution (Symmetric Block Cyclic) was introduced recently [3]. It is valid for specific values of P , of the form either a 2 /2 for an even integer a, or a(a − 1)/2 for any integer a. SBC induces a communication volume lower by a factor of √ 2 than 2DBC, but however remains within a factor of √ 2 of the lower bound [8].…”

On the Arithmetic Intensity of Distributed-Memory Dense Matrix Multiplication Involving a Symmetric Input Matrix (SYMM)

“…Out of the matrix multiplication context, Beaumont et al recently proposed to exploit the symmetry of matrices to enhance the distributed-memory Cholesky factorization of dense symmetric positive definite matrices [8]. The main idea is to rely on an alternative data distribution referred to as symmetric block cyclic (SBC).…”