Novel HPC techniques to batch execution of many variable size BLAS computations on GPUs

Abstract

This paper presents a software framework for solving large numbers of relatively small matrix problems using GPUs. Our approach combines novel and existing HPC techniques to methodically apply performance analysis, kernel design, low-level optimizations, and autotuning to exceed in performance proprietary vendor libraries. As a case study, we discuss the fundamental matrix operations defined by the Basic Linear Algebra Subprograms (BLAS) standard. This case study is significantly important for wide range of applications, including astrophysics, tensor contractions, sparse direct solvers, and others. We provide a generic design that is capable of dealing with problems of different sizes, and handling the irregularity arising from size variations. The developed solution adopts a batched computation scheme, where the same operation is concurrently applied to all matrices within a single computational kernel. The paper discusses the data layout, kernel design, and optimization techniques. We also propose a design scheme that is centralized around matrix-matrix multiplication (GEMM) kernel, so that any improvement on this particular kernel propagates automatically to other routines. Our performance results show significant speedups using a Pascal generation GPU (Tesla P100) against state-of-the-art solutions using cuBLAS, as well as against two 10-core Haswell CPUs running the MKL library. This work is part of the MAGMA library. CCS CONCEPTS •Computing methodologies →Massively parallel algorithms;

Kokkos Kernels: Performance Portable Sparse/Dense Linear Algebra and Graph Kernels

“…The Kokkos Kernels batched BLAS/LAPACK interface provides multiple functor-level interfaces for dense linear algebra (DLA), which is suitable for Kokkos hierarchical parallelism. Unlike other batched BLAS and LAPACK interface, Intel batched GEMM [9], cuBLAS batched GEMM [24], MAGMA batched GEMM [25], we do not provide a frontlevel (or subroutine) interface that launches a streaming parallel kernel. Instead, we provide a functor-level interface that can be used in a Kokkos parallel execution patterns e.g., parallel for, parallel reduce and parallel scan.…”

“…The input arrays contain n random 32-bit integers, with times averaged over 1000 different arrays. Thrust is obviously the fastest for the device-level sort, but Kokkos Kernels gives an average speedup of 1.3x over Kokkos for 2 16 ≤ n ≤ 2 25 .…”

Enabling Highly Efficient Batched Matrix Multiplications on SW26010 Many-core Processor

“…Blocking factors. The blocking factors have significant impact on the performance of matrix multiplications and have been carefully analyzed and tuned in many previous works, such as References [9,40,47]. The evaluation of the blocking factors depends on the methods of parallelization and constraints of LDM capacity and can be used to extract important information such as the on-chip data reuse and the amount of data transfer between the main memory and the LDM.…”

“…As an important extension of the traditional Basic Linear Algebra Subprograms (BLAS) library [20], the new BLAS proposal has already suggested the batched matrix multiplications as an important complement [18]. Recently, vendor-provided libraries such as Intel MKL [1] and NVIDIA cuBLAS [2], and academic research programs such as MAGMA [9], have all added the support of this subroutine. Therefore, it is of great importance to study performance optimization methods for the batched matrix multiplications on state-of-theart hardware platforms.…”

Variable-Size Batched Condition Number Calculation on GPUs

“…Batched routines are typically classified into two subsets: those where all data entities have the same size, and those where the data entities can differ in size (within a range). The latter type of batched routines, usually referred to as "variable-size," are more complicated in design, but offer higher flexibility in terms of target applications [2].…”