Communication-Avoiding Cholesky-QR2 for Rectangular Matrices

Abstract

Scalable QR factorization algorithms for solving least squares and eigenvalue problems are critical given the increasing parallelism within modern machines. We introduce a more general parallelization of the CholeskyQR2 algorithm and show its effectiveness for a wide range of matrix sizes. Our algorithm executes over a 3D processor grid, the dimensions of which can be tuned to trade-off costs in synchronization, interprocessor communication, computational work, and memory footprint. We implement this algorithm, yielding a code that can achieve a factor of Θ(P 1/6 ) less interprocessor communication on P processors than any previous parallel QR implementation. Our performance study on Intel Knights-Landing and Cray XE supercomputers demonstrates the effectiveness of this CholeskyQR2 parallelization on a large number of nodes. Specifically, relative to ScaLAPACK's QR, on 1024 nodes of Stampede2, our CholeskyQR2 implementation is faster by 2.6x-3.3x in strong scaling tests and by 1.1x-1.9x in weak scaling tests.

On the parallel I/O optimality of linear algebra kernels

“…Furthermore, to secure high performance, we carefully tune block sizes and communication routines to maximize the efficiency of local computations such as trsm (triangular solve) and gemm (matrix multiplication). We measure both communication volume and achieved performance of COnf LUX and COnf CHOX and compare them to state-ofthe-art libraries: a vendor-optimized Intel MKL [34], SLATE [28] recent library targeting exascale systems), as well as CANDMC [57,58] and CAPITAL [32,33] (codes based on the asymptotically optimal 2.5D decomposition). In our experiments on the Piz Daint supercomputer, we measure up to 1.6x communication reduction compared to the second-best implementation.…”

“…Matrix factorizations are included in most of linear solvers' libraries. With regard to the parallelization strategy, these libraries may be categorized into three groups: task-based: SLATE [28] (OpenMP tasks), DLAF [35] (HPX tasks), DPLASMA [12] (DaGuE scheduler), or CHAMELEON [3] (StarPU tasks); static 2D parallel: MKL [34], Elemental [53], or Cray LibSci [16]; communicationminimizing 2.5D parallel: CANDMC [57] and CAPITAL [33]. In the last decade, heavy focus was placed on heterogeneous architectures.…”

“…While asymptotically optimal matrix factorizations were proposed, among others, by Ballard et al [7] and Solomonik et al [33,61], we observe two major challenges with the existing approaches: First, the presented algorithms are only asymptotically optimal: the I/O cost of these proposed parallel algorithms can be as high as 7 times the lower bound for LU [61] and up to 16 times for Cholesky [33]. This means that they communicate less than "standard" 2D algorithms like ScaLAPACK [14] only for almost prohibitively large numbers of processors -e.g., according to the LU cost Performance below 3% of hardware peak fastest state-of-the-art library (S=SLATE [28], C=CANDMC [57], M=MKL [34]).…”

“…However, the end of Dennard scaling [22] puts increasing pressure on data movement minimization, as the cost of moving data far exceeds its computation cost, both in terms of power and time [40,63]. Thus, deriving algorithmic I/O lower bounds is a subject of both theoretical analysis [15,36,37] and practical value for developing I/O-efficient schedules [33,59,60].…”

“…Figure 11: Left: measured runtime speedup of COnf CHOX vs.fastest state-of-the-art library (S=SLATE[28], C=CAPITAL[33], M=MKL[34]). Right: COnf CHOX's achieved % of machine peak performance.…”