Sparse LU factorization with partial pivoting on distributed memory machines

Abstract

Abstract-A sparse LU factorization based on Gaussian elimination with partial pivoting (GEPP) is important to many scientific applications, but it is still an open problem to develop a high performance GEPP code on distributed memory machines. The main difficulty is that partial pivoting operations dynamically change computation and nonzero fill-in structures during the elimination process. This paper presents an approach called S* for parallelizing this problem on distributed memory machines. The S* approach adopts static symbolic factorization to avoid run-time control overhead, incorporates 2D L/U supernode partitioning and amalgamation strategies to improve caching performance, and exploits irregular task parallelism embedded in sparse LU using asynchronous computation scheduling. The paper discusses and compares the algorithms using 1D and 2D data mapping schemes, and presents experimental studies on Cray-T3D and T3E. The performance results for a set of nonsymmetric benchmark matrices are very encouraging, and S* has achieved up to 6.878 GFLOPS on 128 T3E nodes. To the best of our knowledge, this is the highest performance ever achieved for this challenging problem and the previous record was 2.583 GFLOPS on shared memory machines [8].

S+: Efficient 2D Sparse LU Factorization on Parallel Machines

“…In the previous work, we show that static factorization does not produce too many fill-ins for most of our test matrices, even for large matrices using a simple matrix ordering strategy (minimum degree ordering) [10,11]. For a few matrices that we have tested, static factorization generates an excessive number of fill-ins.…”

“…Most notably, the recent shared memory implementation of SuperLU has achieved up to 2.58 GFLOPS on 8 Cray C90 nodes [4,5,23]. For distributed memory machines, we proposed an approach that adopts a static symbolic factorization scheme to avoid data structure variation [10,11]. Static symbolic factorization eliminates the runtime overhead of dynamic symbolic factorization with a price of overestimated fill-ins and, thereafter, extra computation [15].…”

“…Since coarsegrain partitioning can reduce available parallelism and produce large submatrices that do not fit into the cache, an upper bound on the supernode size is usually enforced in the L/U supernode partitioning. After the L/U supernode partitioning, each diagonal submatrix is dense, and each nonzero off-diagonal submatrix in the L part contains only dense subrows, and furthermore, each nonzero submatrix in the U part of A contains only dense subcolumns [11]. This is the key to maximize the use of BLAS-3 subroutines [7] in our algorithm.…”

Elimination forest guided 2D sparse LU factorization

“…We examine how the computation-dominating part of the LU algorithm is e ciently implemented using the level of Perform Update2D(k m) for blocks this processor owns (10) if column block m is not factorized and all m's child supernodes have been factorized then (11) Perform Factor(m) for blocks this processor owns (12) endif (13) for j = m + 1 to N (14) if my cno = j mod pc then (15) Perform Update2D(k j) for blocks this processor owns (16) endif (17) endfor (18) endfor There could be several approaches to circumvent the above problem: One approach is to use the mixture of BLAS-1/2/3 routines. If A i k and Ai j have the same row sparse structure, and A k j and Ai j have the same column sparse structure, BLAS-3 GEMM can be directly used to modify Ai j.…”

“…Our previous study 8,10] shows that even with the introduction of extra nonzero elements by static symbolic factorization, the performance of the S sequential code can still be competitive to SuperLU because we are able to use more BLAS-3 operations. Table 3 and the improvement ratio in terms of MFLOPS vary from 16% to 116%, in average more than 50%.…”