Hardware accelerator systems for artificial intelligence and machine learning

Park, Hyunbin; Kim, Shiho

doi:10.1016/bs.adcom.2020.11.005

Cited by 16 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To test the efficacy of our proposed dynamic inference method, DynaTran, we evaluate encoder-only models (because of their high parallelization capabilities [17]) on different tasks. We use BERT-Tiny [26] and BERT-Base [1], two commonly used pre-trained models.…”

Section: A Evaluation Models and Datasetsmentioning

confidence: 99%

“…For mobile platforms, we compare the inference of BERT-Tiny on AccelTran-Edge with off-the-shelf platforms that include Raspberry Pi 4 Model-B [47] that has the Broadcom BCM2711 ARM SoC, Intel Neural Compute Stick (NCS) v2 [48] with its neural processing unit (NPU), and Apple M1 ARM SoC [49] with an 8-core CPU, an 8-core GPU, and 16 GB unified memory on an iPad (for easier evaluations, we performed experiments on a MacBook Pro laptop with the same SoC instead). For serverside platforms, we compare the inference of BERT-Base on AccelTran-Server with a modern NVIDIA A100 GPU (40GB of video RAM) and previously proposed accelerators, namely, OPTIMUS [17], SpAtten [15], and Energon [16]. We chose the maximum batch size possible for each platform, based on its memory capacity.…”

Section: Evaluation Baselinesmentioning

confidence: 99%

“…However, it only exploits sparsity in the attention probabilities, not in all possible multiplication operations in the transformer architecture (details in Section II-B). To tackle this problem, OPTI-MUS [17] uses a set-associative rearranged compressed sparse column (SA-RCSC) format to eliminate ineffectual multiplyand-accumulate (MAC) operations. However, it only exploits sparsity in the weight matrices and not the activations, i.e., the matrices formed from intermediate MAC operations.…”

Section: Introductionmentioning

confidence: 99%

“…Further, these works implement an entire matrix multiplication over an array of processing elements (PEs), which are the basic compute blocks of an accelerator. OPTIMUS [17], with its SA-RCSC sparse matrix format, does not break down the matrices involved into multiple tiles in order to improve hardware utilization. FTRANS [20] and SpAtten [15] break down a matrix-matrix multiplication op-eration into multiple matrix-vector multiplication operations, losing out on data reuse capabilities.…”

Section: Introductionmentioning

confidence: 99%

“…• To efficiently execute DynaTran, we design and implement an ASIC-based architecture called AccelTran. Instead of using traditional encoder-decoder models [17], we leverage recently-proposed encoder-only models [1], thus reducing the critical path by 2× and improving throughput and hardware utilization. Further, unlike previous works [16], AccelTran's dynamic inference pipeline is agnostic to the pre-processed weight pruning strategy.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

Tuli¹,

Jha²

2023

Preprint

View full text Add to dashboard Cite

Self-attention-based transformer models have achieved tremendous success in the domain of natural language processing. Despite their efficacy, accelerating the transformer is challenging due to its quadratic computational complexity and large activation sizes. Existing transformer accelerators attempt to prune its tokens to reduce memory access, albeit with high compute overheads. Moreover, previous works directly operate on large matrices involved in the attention operation, which limits hardware utilization. In order to address these challenges, this work proposes a novel dynamic inference scheme, DynaTran, which prunes activations at runtime with low overhead, substantially reducing the number of ineffectual operations. This improves the throughput of transformer inference. We further propose tiling the matrices in transformer operations along with diverse dataflows to improve data reuse, thus enabling higher energy efficiency. To effectively implement these methods, we propose AccelTran, a novel accelerator architecture for transformers. Extensive experiments with different models and benchmarks demonstrate that DynaTran achieves higher accuracy than the state-of-the-art top-k hardware-aware pruning strategy while attaining up to 1.2× higher sparsity. One of our proposed accelerators, AccelTran-Edge, achieves 330K× higher throughput with 93K× lower energy requirement when compared to a Raspberry Pi device. On the other hand, AccelTran-Server achieves 5.73× higher throughput and 3.69× lower energy consumption compared to the state-of-the-art transformer co-processor, Energon.

show abstract

Section: A Evaluation Models and Datasetsmentioning

confidence: 99%