Shfl-BW

Huang, Guyue; Li, Haoran; Qin, Minghai; Sun, Fei; Ding, Yufei; Xie, Yuan

doi:10.1145/3489517.3530588

Cited by 8 publications

(6 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the synthetic matrices are intended for benchmarking purposes, they have been populated with random values. For each of these inputs sets, different density levels (d ∈ {0.05, 0.10, 0.20, 0.30, 0.50, 0.09}), and pruning configurations of vector-wise [24] (l ∈ {16, 32, 64, 128}) and VENOM [5] (V ∈ {32, 64, 128}) formats were considered.…”

Section: Resultsmentioning

confidence: 99%

“…However, in the same way that happened in dense computation [56], there is an emerging trend of template-based implementations for DL routines. A new wave of third-party kernels has surfaced, providing implementations for different sparse formats, such as Shfl-BW [24] and VENOM [5]. Such solutions offer as configurable parameters the whole set of variables described in Figure 1 such as Thread-Block, Warp, and MMA tile shapes, as well as the Batch size.…”

Section: Emerging Trend: Templated Librariesmentioning

confidence: 99%

“…As an example, for a vector-wise-based kernel [24] and one GPU architecture (i.e., A100-40GB), this is translated into β ≈ 400, which represents a huge number of classes to tune. A detailed study on the impact of the TS design will be made in Section V-B.…”

Section: ) Tuning Space Generatormentioning

confidence: 99%

See 2 more Smart Citations

STuning-DL: Model-Driven Autotuning of Sparse GPU Kernels for Deep Learning

Castro,

Andrade,

Fraguela

2024

IEEE Access

View full text Add to dashboard Cite

The relentless growth of modern Machine Learning models has spurred the adoption of sparsification techniques to simplify their architectures and reduce the computational demands. Network pruning has demonstrated success in maintaining original network accuracy while shedding significant portions of the original weights. However, leveraging this sparsity efficiently remains challenging due to computational irregularities, particularly in GPU kernels. A new trend of template-based GPU kernels for semi-structured sparsity shows promise in efficiency but lacks autotuning capabilities to adapt to input dynamics, often underperforming in scenarios where they have not been meticulously hand-tuned. We present STuning-DL, the first pruning-aware autotuner for third-party template-based implementations enabling efficient optimization of sparse kernels for Deep Learning, spanning from high-level aspects (CUDA C++ level) down to GPU-native instructions specifics (assembly-level). STuning-DL tunes and optimizes at run-time sparse kernels' performance for each input problem, yielding speedups of up to 5.42× on NVIDIA T4-16GB and up to 3.6× on NVIDIA A100-40GB GPU in sparse matrices from real world models compared to existing heuristics from sparse libraries like cuSparse and cuSparseLt.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Emerging Trend: Templated Librariesmentioning

confidence: 99%

See 1 more Smart Citation

STuning-DL: Model-Driven Autotuning of Sparse GPU Kernels for Deep Learning

Castro,

Andrade,

Fraguela

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…However, achieving substantial gains in throughput or training time on SIMD and systolic array based architectures like GPUs and TPUs remains challenging. Notably, the SIMD nature of GPUs, including the incorporation of Tensor Cores, presents limitations for efficiently accelerating sparse operations (Gale et al 2020;Huang et al 2022). Consequently, achieving benefits from sparse matrix matrix multiplication (SpMM) in the absence of specific sparsity patterns proves difficult, except for extreme sparsity levels of around 99% and higher (Hoefler et al 2021).…”

Section: Related Workmentioning

confidence: 99%

“…Consequently, achieving benefits from sparse matrix matrix multiplication (SpMM) in the absence of specific sparsity patterns proves difficult, except for extreme sparsity levels of around 99% and higher (Hoefler et al 2021). To address this, techniques have emerged to introduce and exploit structure within sparse matrices (Huang et al 2022;Mishra et al 2021;Wang 2020). For instance, NVIDIA's 2:4 sparsity support optimally utilizes tensor cores (Mishra et al 2021).…”

Section: Related Workmentioning

confidence: 99%

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

Finkbeiner,

Gmeinder,

Pupilli

et al. 2024

AAAI

View full text Add to dashboard Cite

Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today's prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation though time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient, non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.

show abstract

A comprehensive review of model compression techniques in machine learning

Dantas,

Sabino da Silva,

Cordeiro

et al. 2024

Appl Intell

View full text Add to dashboard Cite

This paper critically examines model compression techniques within the machine learning (ML) domain, emphasizing their role in enhancing model efficiency for deployment in resource-constrained environments, such as mobile devices, edge computing, and Internet of Things (IoT) systems. By systematically exploring compression techniques and lightweight design architectures, it is provided a comprehensive understanding of their operational contexts and effectiveness. The synthesis of these strategies reveals a dynamic interplay between model performance and computational demand, highlighting the balance required for optimal application. As machine learning (ML) models grow increasingly complex and data-intensive, the demand for computational resources and memory has surged accordingly. This escalation presents significant challenges for the deployment of artificial intelligence (AI) systems in real-world applications, particularly where hardware capabilities are limited. Therefore, model compression techniques are not merely advantageous but essential for ensuring that these models can be utilized across various domains, maintaining high performance without prohibitive resource requirements. Furthermore, this review underscores the importance of model compression in sustainable artificial intelligence (AI) development. The introduction of hybrid methods, which combine multiple compression techniques, promises to deliver superior performance and efficiency. Additionally, the development of intelligent frameworks capable of selecting the most appropriate compression strategy based on specific application needs is crucial for advancing the field. The practical examples and engineering applications discussed demonstrate the real-world impact of these techniques. By optimizing the balance between model complexity and computational efficiency, model compression ensures that the advancements in AI technology remain sustainable and widely applicable. This comprehensive review thus contributes to the academic discourse and guides innovative solutions for efficient and responsible machine learning practices, paving the way for future advancements in the field. Graphical abstract

show abstract

Shfl-BW

Cited by 8 publications

References 3 publications

STuning-DL: Model-Driven Autotuning of Sparse GPU Kernels for Deep Learning

STuning-DL: Model-Driven Autotuning of Sparse GPU Kernels for Deep Learning

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

A comprehensive review of model compression techniques in machine learning

Contact Info

Product

Resources

About