High Performance Multi-GPU SpMV for Multi-component PDE-Based Applications

Abstract

Abstract. Leveraging optimization techniques (e.g., register blocking and double buffering) introduced in the context of KBLAS, a Level 2 BLAS high performance library on GPUs, the authors implement dense matrix-vector multiplications within a sparse-block structure. While these optimizations are important for high performance dense kernel executions, they are even more critical when dealing with sparse linear algebra operations. The most time-consuming phase of many multicomponent applications, such as models of reacting flows or petroleum reservoirs, is the solution at each implicit time step of large, sparse spatially structured or unstructured linear systems. The standard method is a preconditioned Krylov solver. The Sparse Matrix-Vector multiplication (SpMV) is, in turn, one of the most time-consuming operations in such solvers. Because there is no data reuse of the elements of the matrix within a single SpMV, kernel performance is limited by the speed at which data can be transferred from memory to registers, making the bus bandwidth the major bottleneck. On the other hand, in case of a multi-species model, the resulting Jacobian has a dense block structure. For contemporary petroleum reservoir simulations, the block size typically ranges from three to a few dozen among different models, and still larger blocks are relevant within adaptively model-refined regions of the domain, though generally the size of the blocks, related to the number of conserved species, is constant over large regions within a given model. This structure can be exploited beyond the convenience of a block compressed row data format, because it offers opportunities to hide the data motion with useful computations. The new SpMV kernel outperforms existing state-of-the-art implementations on single and multi-GPUs using matrices with dense block structure representative of porous media applications with both structured and unstructured multi-component grids.

A framework for dense triangular matrix kernels on various manycore architectures

“…KAUST BLAS (KBLAS) is an open‐source library that provides highly optimized implementations for a subset of BLAS routines on NVIDIA GPUs as well as x86 architectures . In particular, the authors have already demonstrated significant performance gains for IP TRSM and TRMM against cuBLAS IP and MAGMA OOP implementations on a single NVIDIA GPU .…”

Performance optimization of Sparse Matrix‐Vector Multiplication for multi‐component PDE‐based applications using GPUs

“…However, this distribution, which has been used in [2], does not work well for all matrices, unless the row lengths are balanced, as proposed in Section 5.5. Block rows of the matrix, preferably reordered according to their lengths, will be distributed among GPUs in a 1D cyclic manner.…”

“…Such matrices are not necessarily block-sparse; however, we are interested in their structures, as inherited from spatial discretization. KSPARSE using BSR format [2]. Such approach enables us to test the performance of the proposed kernels against a wide range of sparsity patterns.…”

“…Table II shows some properties of the selected matrices. Such structures create a high load imbalance for the original KSPARSE kernel [2], due to the fact that it assigns an entire row to the same group of threads. Figure 9 shows the structures of these matrices.…”

“…(2) Instead of matrices with synthetic structure, the performance reported in this paper is based on matrices with structures arising from real applications. Based on the histogram of the nonzero row lengths of these matrices, we show that the performance of the original kernel [2] may be unacceptable and propose a modification that enables a much better performance. This gives insight on the performance of the proposed kernel on more realistic structures.…”