The Design and Performance of Batched BLAS on Modern High-Performance Computing Systems

Abstract

A current trend in high-performance computing is to decompose a large linear algebra problem into batches containing thousands of smaller problems, that can be solved independently, before collating the results. To standardize the interface to these routines, the community is developing an extension to the BLAS standard (the batched BLAS), enabling users to perform thousands of small BLAS operations in parallel whilst making efficient use of their hardware. We discuss the benefits and drawbacks of the current batched BLAS proposals and perform a number of experiments, focusing on a general matrix-matrix multiplication (GEMM), to explore their affect on the performance. In particular we analyze the effect of novel data layouts which, for example, interleave the matrices in memory to aid vectorization and prefetching of data. Utilizing these modifications our code outperforms both MKL1 CuBLAS2 by up to 6 times on the self-hosted Intel KNL (codenamed Knights Landing) and Kepler GPU architectures, for large numbers of double precision GEMM operations using matrices of size 2 × 2 to 20 × 20.The authors would like to thank The University of Tennessee for the use of their computational resources. This research was funded in part from the European Union’s Horizon 2020 research and innovation programme under the NLAFET grant agreement No. 671633.Peer ReviewedPostprint (published version

NVIDIA Tensor Core Programmability, Performance & Precision

“…Many HPC applications rely on the solution of several small-size matrix multiplications in parallel [22]. One example is the Nek5000 CFD application that uses small-size matrix multiplies for each spectral element resulting from the semi-spectral discretization [23], [24].…”

Enabling Highly Efficient Batched Matrix Multiplications on SW26010 Many-core Processor

“…There are three typical data storage formats for matrix multiplications: the P2P format, the strided format, and the interleaved format [19,24,30]. The P2P format uses arrays whose elements are pointers to memory locations containing matrices, and the pointer arrays are passed as kernel parameters.…”

“…In the past few years, the batched matrix multiplications have drawn increasingly more attention in both the industry [1,2] and the academy [8,19,30]. With the rapid development of high-performance computing, many-core-based architectures that rely on many lightweight computing cores and a deep memory hierarchy are becoming an important solution in designing modern supercomputers.…”

“…This trend gives rise to the applications of batched matrix multiplications, which can be frequently found in, e.g., quantum chemistry [13], astrophysics [39], metabolic networks [32], computational fluid dynamics [44], domain decomposition solvers [10], tensor computations [45], and deep learning [6,11]. It has been proven that in these applications, the performance could be greatly improved by exploiting batched computations of small matrix multiplications [8,19,37]. As an important extension of the traditional Basic Linear Algebra Subprograms (BLAS) library [20], the new BLAS proposal has already suggested the batched matrix multiplications as an important complement [18].…”

A Guide for Achieving High Performance with Very Small Matrices on GPU: A Case Study of Batched LU and Cholesky Factorizations

“…This kind of design might be interesting for a GEMV or TRSV type of operation, where the matrix is read only once. Recent studies on optimized batched BLAS kernels designed for multicore architectures have shown promising results over the classical approach of solving one problem per core at a time [15], [16].…”