Hierarchical approach for deriving a reproducible unblocked LU factorization

Abstract

We propose a reproducible variant of the unblocked LU factorization for graphics processor units (GPUs).For this purpose, we build upon Level-1/2 BLAS kernels that deliver correctly-rounded and reproducible results for the dot (inner) product, vector scaling, and the matrix-vector product. In addition, we draw a strategy to enhance the accuracy of the triangular solve via iterative refinement. Following a bottom-up approach, we finally construct a reproducible unblocked implementation of the LU factorization for GPUs, which accommodates partial pivoting for stability and can be eventually integrated in a high performance and stable algorithm for the (blocked) LU factorization.The IEEE 754 standard, created in 1985 and then revised in 2008, has led to a considerable enhancement in the reliability of numerical computations by rigorously specifying the properties of floating-point arithmetic. This standard is now adopted by most processors, thus leading to a much better portability of numerical applications.Exascale computing (10 18 operations per second) is likely to be reached within a decade. For the type of systems yielding such performance rate, getting accurate and reproducible 1 results in floating-point arithmetic will represent two considerable challenges [9,28]. Reproducibility is also an important and useful property when debugging and checking the correctness of codes as well as for legal issues.Email addresses: riakymch@kth.se (Roman Iakymchuk), stef.graillat@lip6.fr (Stef Graillat), david.defour@univ-perp.fr (David Defour), quintana@uji.es (Enrique S. Quintana-Ortí) 1 By accuracy, we mean the relative error between the exact result and the computed result. We define reproducibility as the ability to obtain a bit-wise identical floating-point result from multiple runs of the code on the same input data.

Reproducibility strategies for parallel Preconditioned Conjugate Gradient

“…While ExSUM covers wide range of architectures as well as distributed-memory clusters, the other routines primarily target GPUs. Exploiting the modular and hierarchical structure of linear algebra algorithms, the ExBLAS approach was applied to construct reproducible LU factorizations with partial pivoting [8].…”

“…The later can also be viewed as switching to fixed-precision computations. Additionally, the bit-wise reproducibility can get costly with the overhead of at least 8 % for parallel reduction [6,7], up to 2x-4x for matrix-vector product [8], and more than 10x for matrix-matrix multiplication [9].…”

Reproducibility, accuracy and performance of the Feltor code and library on parallel computer architectures

“…Our basic assumption is that, if these elementary functions are reproducible, then all algorithms and simulations implemented with them are reproducible. This assumption follows our theoretical and practical studies [37] of the unblocked LU factorization with partial pivoting, which underneath is entirely build upon the BLAS routines. The first step to realize our goal incorporates the correctly rounded and reproducible parallel reduction from the ExBLAS library into Feltor.…”

Reproducibility of parallel preconditioned conjugate gradient in hybrid programming environments

“…While ExSUM covers wide range of architectures as well as distributed-memory clusters, the other routines primarily target GPUs. Exploiting the modular and hierarchical structure of linear algebra algorithms, the ExBLAS approach was applied to construct reproducible LU factorizations with partial pivoting Iakymchuk et al (2019b).…”

“…These modifications are necessary to preserve every bit of information (both result and error) Collange et al (2015) or, alternatively, to cut off some parts of the data and operate on the remaining most significant parts Mukunoki et al (2020); Demmel and Nguyen (2015). Furthermore, the bit-wise reproducibility can become expensive with the overhead of at least 8% for parallel reduction Collange et al (2015); Demmel and Nguyen (2015), up to 2x–4x for matrix-vector product Iakymchuk et al (2019b), and more than 10x for matrix–matrix multiplication Iakymchuk et al (2016). In this paper, we aim to revisit reproducibility and raise its appeal through reducing its negative impact on performance and minimizing changes to both the algorithm and its building blocks.…”