I/O-Optimal Algorithms for Symmetric Linear Algebra Kernels

Abstract

In this paper, we consider two fundamental symmetric kernels in linear algebra: the Cholesky factorization and the symmetric rankupdate (SYRK), with the classical three nested loops algorithms for these kernels. In addition, we consider a machine model with a fast memory of size and an unbounded slow memory. In this model, all computations must be performed on operands in fast memory, and the goal is to minimize the amount of communication between slow and fast memories. As the set of computations is fixed by the choice of the algorithm, only the ordering of the computations (the schedule) directly influences the volume of communications.We prove lower bounds of 1 3 √ 2 3 √ for the communication volume of the Cholesky factorization of an × symmetric positive definite matrix, and of 1 √ 2 2 √ for the SYRK computation of A • A T , where A is an × matrix. Both bounds improve the best known lower bounds from the literature by a factor √ 2. In addition, we present two out-of-core, sequential algorithms with matching communication volume: TBS for SYRK, with a volume of 1 √ 2 2 √ + O ( log ), and LBC for Cholesky, with a volume of 1 3 √ 2 3 √ + O ( 5/2). Both algorithms improve over the best known algorithms from the literature by a factor √ 2, and prove that the leading terms in our lower bounds cannot be improved further. This work shows that the operational intensity of symmetric kernels like SYRK or Cholesky is intrinsically higher (by a factor √ 2) than that of corresponding non-symmetric kernels (GEMM and LU factorization).

Symmetric Block-Cyclic Distribution: Fewer Communications Leads to Faster Dense Cholesky Factorization

“…Based on a discrete version of classical Loomis-Whitney projection argument developed in [12], Olivry et al establish in [8], that Cholesky factorization requires at least n 3 6 √ M data transfers. A specific analysis adapted to the symmetric nature of this factorization allowed Beaumont et al [13] to further improve the bound and obtain an optimal value of…”

“…In 2009, Béreux [14] proposed a sequential out-of-core Cholesky algorithm with "narrow blocks" that performs at most n 3 3 √ M + O(n 2 ) data transfers. The authors of [13] have improved this result with an algorithm which performs…”

“…The SBC distribution is an adequate adaptation of Béreux's algorithm for the parallel distributed Cholesky factorization. A recent result [13] shows that Béreux's algorithm is actually not optimal for Cholesky: it is possible to achieve √ 2M arithmetic intensity. However, adapting the algorithm of [13] to the parallel setting is beyond the scope of this work.…”

“…A recent result [13] shows that Béreux's algorithm is actually not optimal for Cholesky: it is possible to achieve √ 2M arithmetic intensity. However, adapting the algorithm of [13] to the parallel setting is beyond the scope of this work.…”

Data Distribution Schemes for Dense Linear Algebra Factorizations on Any Number of Nodes

“…The SBC distribution (Symmetric Block Cyclic) was introduced recently [3]. It is valid for specific values of P , of the form either a 2 /2 for an even integer a, or a(a − 1)/2 for any integer a. SBC induces a communication volume lower by a factor of √ 2 than 2DBC, but however remains within a factor of √ 2 of the lower bound [8]. In this paper, we propose an extension of this symmetric distribution to all possible values of P .…”