DOTA: detect and omit weak attentions for scalable transformer acceleration

Abstract

Transformer Neural Networks have demonstrated leading performance in many applications spanning over language understanding, image processing, and generative modeling. Despite the impressive performance, long-sequence Transformer processing is expensive due to quadratic computation complexity and memory consumption of self-attention. In this paper, we present DOTA, an algorithmarchitecture co-design that effectively addresses the challenges of scalable Transformer inference. Based on the insight that not all connections in an attention graph are equally important, we propose to jointly optimize a lightweight Detector with the Transformer model to accurately detect and omit weak connections during runtime. Furthermore, we design a specialized system architecture for end-to-end Transformer acceleration using the proposed attention detection mechanism. Experiments on a wide range of benchmarks demonstrate the superior performance of DOTA over other solutions. In summary, DOTA achieves 152.6× and 4.5× performance speedup and orders of magnitude energy-efficiency improvements over GPU and customized hardware, respectively. CCS CONCEPTS• Computer systems organization → Neural networks; • Computing methodologies → Machine learning approaches.

GCoD: Graph Convolutional Network Acceleration via Dedicated Algorithm and Accelerator Co-Design

“…Our ViTCoD split and conquer algorithm exhibits a great potential in both reducing the dominate attention computations and alleviating the irregularity of the resulting sparse attention masks. However, this potential cannot be fully exploited by existing Transformer accelerators [21], [27], [39] due to the fact that (1) they are designed for dynamic sparse attention which requires both on-the-fly mask generation and highly reconfigurable architecture supports, both of which require nontrivial overheads, and (2) they are not dedicated for processing the enforced two distinct workloads, i.e., denser and sparser patterns, from our ViTCoD algorithm. As such, our ViTCoD accelerator is motivated to exploit the new opportunities i.e., fixed and structurally sparse patterns, resulting from ViTCoD algorithm to boost ViTs' inference efficiency.…”

“…Baselines: To benchmark ViTCoD with SOTA attention accelerators, we consider a total of five baselines, including three general platforms: CPU (Intel Xeon Gold 6230R), EdgeGPU (Nvidia Jetson Xavier NX), and GPU (Nvidia 2080Ti), and two attention accelerators: SpAtten [39] and Sanger [21]. Note that when benchmarking with GPUs w/ larger batch size, we scale up the accelerators' hardware resource to have a comparable peak throughput for a fair comparison following [27]. Metrics: We 0.9 0.9 1.9 evaluate all platforms in terms of latency speedups and energy efficiency.…”

“…SpAtten [39] structurally removes unnecessary attention heads and input tokens, which is therefore coarse-grained and leads to a low achievable sparsity ratio; Sanger [21] adopts low precision Q and K vectors for estimating the sparse attention masks, which are then packed and split to be more regular and friendly supported by a reconfigurable architecture. DOTA [27] considers both low precision and low rank linear transformation to predict the sparse attention masks, and explores token-level parallelism and out-of-order execution for locality-aware computing. All above works focus on NLP Transformers, and thus require dynamic and input-dependent sparse masks prediction.…”

“…accounts for over 50% of the total latency measured on a TITAN Xp GPU [39]; This percentage increases to 69% for LeViT-128 [9] when measured on an EdgeGPU [23]. To alleviate the bottleneck complexity of self-attention modules, sparse attention techniques have emerged as a promising solution and been considered by both algorithm [3], [36], [44] and hardware acceleration [11], [21], [27], [39] works. Despite their great promise, existing sparse attention accelerators or algorithm-accelerator co-design works (e.g., Sanger [21]) focus on accelerating NLP Transformers, and adopt hardware designs with on-the-fly sparse attention prediction and high reconfigurability in order to handle the varying number of input tokens in NLP.…”

ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention

“…Then, the sparse attention matrix with reduced entries goes through the softmax operation after which it is multiplied by a dense value matrix. Many works in algorithm [2], [10], [12], [13], [48], [49] and hardware [21], [22], [28], [33], [40] have been proposed to implement such sparse attentions for NLPbased Transformer models by efficiently tackling various static and dynamic sparse patterns.…”