TileSpMV: A Tiled Algorithm for Sparse Matrix-Vector Multiplication on GPUs

Niu, Yuyao; Lu, Zhengyang; Dong, Meichen; Jin, Zheming; Liu, Weifeng; Tan, Guangming

doi:10.1109/ipdps49936.2021.00016

Cited by 45 publications

(12 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, it is important to optimize these operations for high gains. Unlike previous works that perform matrix multiplications directly using large MAC units, we propose using tiled matrix multiplication (primarily employed by modern GPUs [33]). Tiling the operations helps with better utilization of resources and enables massive parallelization.…”

Section: B the Acceltran Simulatormentioning

confidence: 99%

AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

Tuli¹,

Jha²

2023

Preprint

View full text Add to dashboard Cite

Self-attention-based transformer models have achieved tremendous success in the domain of natural language processing. Despite their efficacy, accelerating the transformer is challenging due to its quadratic computational complexity and large activation sizes. Existing transformer accelerators attempt to prune its tokens to reduce memory access, albeit with high compute overheads. Moreover, previous works directly operate on large matrices involved in the attention operation, which limits hardware utilization. In order to address these challenges, this work proposes a novel dynamic inference scheme, DynaTran, which prunes activations at runtime with low overhead, substantially reducing the number of ineffectual operations. This improves the throughput of transformer inference. We further propose tiling the matrices in transformer operations along with diverse dataflows to improve data reuse, thus enabling higher energy efficiency. To effectively implement these methods, we propose AccelTran, a novel accelerator architecture for transformers. Extensive experiments with different models and benchmarks demonstrate that DynaTran achieves higher accuracy than the state-of-the-art top-k hardware-aware pruning strategy while attaining up to 1.2× higher sparsity. One of our proposed accelerators, AccelTran-Edge, achieves 330K× higher throughput with 93K× lower energy requirement when compared to a Raspberry Pi device. On the other hand, AccelTran-Server achieves 5.73× higher throughput and 3.69× lower energy consumption compared to the state-of-the-art transformer co-processor, Energon.

show abstract

Section: B the Acceltran Simulatormentioning

confidence: 99%

AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

Tuli¹,

Jha²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Sparse matrix multiplication (SpMM) is an important computation pattern in Deep Learning domains. Plenty of efforts have been invested in optimization SpMM software on GPUs over the past years [1,3,4,5,17,19,27,37,41,43]. Although significant progresses have been made, the performance of SpMM software on GPUs are not satisfied in terms of the hardware throughput bound.…”

Section: Sparse Fmamentioning

confidence: 99%

Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numerical Behaviors

Sun¹,

Li²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Tensor Cores have been an important unit to accelerate Fused Matrix Multiplication Accumulation (MMA) in all NVIDIA GPUs since Volta Architecture. To program Tensor Cores, users have to use either legacy wmma APIs or current mma APIs. Legacy wmma APIs are more easy-to-use but can only exploit limited features and power of Tensor Cores. Specifically, wmma APIs support fewer operand shapes and can not leverage the new sparse matrix multiplication feature of the newest Ampere Tensor Cores. However, the performance of current programming interface has not been well explored. Furthermore, the computation numeric behaviours of low-precision floating points (TF32, BF16 and FP16) supported by newest Ampere Tensor Cores are also mysterious. In this paper, we explore the throughput and latency of current programming APIs. We also intuitively study the numeric behaviours of Tensor Cores MMA and profile the intermediate operations including multiplication, addition of inner product and addition of accumulation.

show abstract

“…Researchers have proposed many approaches to improve the performance of SpGEMM on GPUs [11]- [16]. We introduce several approaches that use the row-wise and two-phase methods to perform SpGEMM on GPUs.…”

Section: Related Workmentioning

confidence: 99%

“…The extensive use of SpGEMM in real-world applications has led to the development of several SpGEMM libraries on CPUs [7]- [10], GPUs [11]- [16], and accelerators [17]- [20], targeting high-performance computing. This paper focuses on developing a high-performance SpGEMM library on GPUs.…”

Section: Introductionmentioning

confidence: 99%

OpSparse: A Highly Optimized Framework for Sparse General Matrix Multiplication on GPUs

Guan

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Sparse general matrix multiplication (SpGEMM) is an important and expensive computation primitive in many real-world applications. Due to SpGEMM's inherent irregularity and the vast diversity of its input matrices, developing high-performance SpGEMM implementation on modern processors such as GPUs is challenging. The state-of-the-art SpGEMM libraries (i.e., nsparse and spECK) adopt several algorithms to tackle the challenges of global load balance, local load balance, and allocation of the result matrix. While these libraries focus on the high-level algorithm design for SpGEMM, they neglect several low-level architecture-specific optimizations, which causes inefficient implementations in their libraries. In this paper, we classify their inefficient implementations into several categories. Based on our observations, we propose a highly optimized SpGEMM library called OpSparse. The optimizations in OpSparse include 1) optimizing the binning method by improving the utilization of the shared memory, 2) optimizing the hashing method by reducing the access to the hash table, 3) improving the trade-off between hash collision rate and hardware utilization in the hashing method by setting appropriate binning ranges, 4) reducing the overheads of global memory utilization by minimizing the global memory usage of the metadata, and 5) improving the execution parallelism by overlapping global memory allocation with kernel execution. Performance evaluations with 26 commonly used matrices on an Nvidia Tesla V100 GPU show that OpSparse achieves on average 7.35× (up to 27.8×), 1.43× (up to 1.81×), and 1.52× (up to 2.04×) speedups over three state-of-the-art SpGEMM libraries: cuSPARSE , nsparse, and spECK, respectively.INDEX TERMS Sparse general matrix multiplication, SpGEMM, GPU, high-performance computing.

show abstract

TileSpMV: A Tiled Algorithm for Sparse Matrix-Vector Multiplication on GPUs

Cited by 45 publications

References 54 publications

AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numerical Behaviors

OpSparse: A Highly Optimized Framework for Sparse General Matrix Multiplication on GPUs

Contact Info

Product

Resources

About