A GEMM interface and implementation on NVIDIA GPUs for multiple small matrices

Abstract

We present an interface and an implementation of the General Matrix Multiply (GEMM) routine for multiple small matrices processed simultaneously on NVIDIA graphics processing units (GPUs). We focus on matrix sizes under 16. The implementation can be easily extended to larger sizes. For single precision matrices, our implementation is 30% to 600% faster than the batched cuBLAS implementation distributed in the CUDA Toolkit 5.0 on NVIDIA Tesla K20c. For example, we obtain 104 GFlop/s and 216 GFlop/s when multiplying 100,000 independent matrix pairs of size 10 and 16, respectively. Similar improvement in performance is obtained for other sizes, in single and double precision for real and complex types, and when the number of matrices is smaller. Apart from our implementation, our different function interface also plays an important role in the improved performance. Applications of this software include Finite Element computation on GPUs.

Enabling Highly Efficient Batched Matrix Multiplications on SW26010 Many-core Processor

“…There are three typical data storage formats for matrix multiplications: the P2P format, the strided format, and the interleaved format [19,24,30]. The P2P format uses arrays whose elements are pointers to memory locations containing matrices, and the pointer arrays are passed as kernel parameters.…”

“…For example, Dongarra et al [19] compared different interfaces and data storage formats for the batched matrix multiplications, which were also discussed by us in Section 4.4. Jhurani et al [30] proposed an interface by adding a second leading dimension and implemented batched square matrix multiplications of sizes under 16 on the GPUs. In this work, we have applied a similar interface in our implementation with the strided data storage format.…”

“…In the past few years, the batched matrix multiplications have drawn increasingly more attention in both the industry [1,2] and the academy [8,19,30]. With the rapid development of high-performance computing, many-core-based architectures that rely on many lightweight computing cores and a deep memory hierarchy are becoming an important solution in designing modern supercomputers.…”

Optimization Method to Reduce Matrices Multiplications in the Context of CUDA

“…Chittampally Vasanth Raja, Srinivas Balasubramanian and Prakash S aghavendra made a full study heterogeneous highly parallel implementation of matrix exponentiation using GPU [1]. In [2], the authors provided an interface and a general matrix routine Implementation for multiplying small matrices that were simultaneously processed on GPU. Their matrix size was less than 16.…”

“…A GPU uses shared memory to reduce memory latency. In a CPU, workloads do not require a lot of memory access and data is brought where as in the GPU there is a lot of memory access and the bandwidth is developed very well [2]. A GPU is a pile of parallel co-processors.…”