Low synchronization Gram–Schmidt and generalized minimal residual algorithms

Abstract

Summary The Gram–Schmidt process uses orthogonal projection to construct the A = QR factorization of a matrix. When Q has linearly independent columns, the operator P = I − Q(QTQ)−1QT defines an orthogonal projection onto Q⊥. In finite precision, Q loses orthogonality as the factorization progresses. A family of approximate projections is derived with the form P = I − QTQT, with correction matrix T. When T = (QTQ)−1, and T is triangular, it is postulated that the best achievable orthogonality is 𝒪(ε)κ(A). We present new variants of modified (MGS) and classical Gram–Schmidt algorithms that require one global reduction step. An interesting form of the projector leads to a compact WY representation for MGS. In particular, the inverse compact WY MGS algorithm is equivalent to a lower triangular solve. Our main contribution is to introduce a backward normalization lag into the compact WY representation, resulting in a 𝒪(ε)κ([r0,AVm]) stable Generalized Minimal Residual Method (GMRES) algorithm that requires only one global reduce per iteration. Further improvements in performance are achieved by accelerating GMRES on GPUs.

A survey of numerical linear algebra methods utilizing mixed-precision arithmetic

“…Although somewhat outside the scope of this review, we can demonstrate that it is possible to modify the Gratton et al (2020) analysis based on the inverse compact WY form of the MGS algorithm, introduced by Świrydowicz et al (2020). Rather than treat all of the inner products in the MGS-GMRES algorithm equally, consider the strictly upper triangular matrix U=LT from the loss of orthogonality relations.…”

“…Barlow then extends this approach to the block compact WY form of MGS; see also the technical report by Sun (1996). The contribution by Świrydowicz et al (2020) was to note that there exists an inverse compact WY representation for MGS—having the projector P with lower triangular correction matrix T :…”

Performance of Low Synchronization Orthogonalization Methods in Anderson Accelerated Fixed Point Solvers

“…In the case of CGS, CGS-2 (classical Gram-Schmidt with reorthogonalization) corrects the projection by reorthogonalizing the vectors of Q and thereby reduces the loss of orthogonality to O(ε) [10,20]. The form of the correction matrix for this algorithm was derived in [23] recently introduced, requiring a triangular solve instead of the recursive construction of the correction matrix T from a block-triangular inverse (presented in Section 3.2). The required number of dot products per iteration is effectively reduced to one for GMRES and s-step Krylov solvers [23,25].…”

“…The form of the correction matrix for this algorithm was derived in [23] recently introduced, requiring a triangular solve instead of the recursive construction of the correction matrix T from a block-triangular inverse (presented in Section 3.2). The required number of dot products per iteration is effectively reduced to one for GMRES and s-step Krylov solvers [23,25]. The inverse compact W Y algorithm maintains O(ε)κ(A) loss of orthogonality.…”

“…performs multiple dot products simultaneously or computes an equivalent QR update at a lower parallel cost. Recent research by Świrydowicz et al [23] and Bielich et al [4] introduced low synchronization algorithms for MGS, CGS-2 and GMRES [22], and because AA is equivalent to GMRES when solving systems of linear equations [24], it is natural to consider leveraging these advances in AA.…”