Unstructured Weight Pruning in Variability-Aware Memristive Crossbar Neural Networks

Aswani, AR; Chithra, R; James, Alex Pappachen

doi:10.1109/iscas48785.2022.9937284

Cited by 5 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While these techniques are typically not the most effective in literature, they de facto nullify the necessity of designing ad hoc implementations for the pruned model, resulting in totally hardware-friendly methods. In contrast, non-structured pruning methods [18], [19] allow the removal of single interconnections (and, consequentially, of the related weights), granting more degrees of freedom 3 Many non-linear strategies have also been tested to transition from β = 1 to β = 0 and each of them has shown a different impact on the trend of the validation accuracy during training. However, every strategy seems to yield the same final accuracy.…”

Section: Pruning Techniques Classificationmentioning

confidence: 99%

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

Prono,

Bich,

Boretti

et al. 2024

Preprint

View full text Add to dashboard Cite

The growing interest in Internet of Things and mobile Artificial Intelligence applications is pushing the investigation on Deep Neural Networks (DNNs) that can operate at the edge using low-resources/energy devices. To obtain such a goal, several pruning techniques have been proposed in the literature. They aim to reduce the number of interconnections - and consequently the size, and the corresponding computing and storage requirements - of DNNs that traditionally rely on classic Multiply-and-ACcumulate (MAC) neurons. In this work, we propose a novel neuron structure based on a Multiply-And-Max/min (MAM) map-reduce paradigm, and we show that by exploiting this new paradigm it is possible to build naturally and aggressively prunable DNN layers, with a negligible loss in performance. This novel structure allows a greater interconnection sparsity when compared to classic MAC-based DNN layers. Moreover, most of the already existing state-of-the-art pruning techniques can be used with MAM layers with little to no changes. To test the pruning performance of MAM, we employ different models - AlexNet, VGG-16 and the more recent ViT-B/16 - and different computer vision datasets - CIFAR-10, CIFAR-100 and ImageNet-1K. Multiple pruning approaches are applied, ranging from single-shot methods to training-dependent and iterative techniques. As a notable example, we test MAM on the ViT-B/16 model fine-tuned on the ImageNet-1K task and apply one-shot gradient-based pruning. We remove interconnections until the model experiences a 3% decrease in accuracy. While MAC-based layers need at least 56.16% remaining interconnections, MAM-based layers achieve the same accuracy with only 0.04%.

show abstract

Section: Pruning Techniques Classificationmentioning

confidence: 99%

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

Prono,

Bich,

Boretti

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…While these techniques are typically not the most effective in literature, they de facto nullify the necessity of designing ad hoc implementations for the pruned model, resulting in totally hardware friendly methods. Conversely, non-structured pruning methods [15], [16] allow the removal of single interconnections (and, consequentially, of the related weights), granting more degrees of freedom to the pruning process and resulting in better compression results. Nonetheless, this comes with an inhardware computational/memory overhead due to the encoding of the resulting sparse weights matrices [17]- [19].…”

Section: Pruning Techniques Classificationmentioning

confidence: 99%

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

Prono¹,

Bich²,

Mangia³

et al. 2023

Preprint

View full text Add to dashboard Cite

The growing interest in Internet of Things and mobile Artificial Intelligence applications is pushing the investigation on Deep Neural Networks (DNNs) that can operate at the edge using low-resources/energy devices. To obtain such a goal, several pruning techniques have been proposed in the literature. They aim to reduce the number of interconnections -- and consequently the size, and the corresponding computing and storage requirements -- of a DNN relying on classic Multiply-and-ACcumulate (MAC) neurons. In this work, we propose a novel neurons structure based on a Multiply-And-Max/min (MAM) map-reduce paradigm, and we show that by exploiting this new paradigm it is possible to build naturally and aggressively prunable DNN layers, with a negligible loss in performance. In fact, this novel structure allows a greater interconnection sparsity when compared to classic MAC based DNN layers. Moreover, most of the already existing state-of-the-art pruning techniques can be used with MAM layers with little to no changes. As an example, by applying one-shot pruning to a VGG-16 model trained on the ImageNet task, fully connected MAM-based layers need only 0.04% of the total number of interconnections while MAC-based layers need at least 4.33%, with a Top-1 accuracy loss of 3% compared to the maximum achieved accuracy. Additionally, we test Lottery Ticket iterative pruning on AlexNet with CIFAR-100 task. With 0.02% remaining interconnections, the MAC-based model requires 10 training iterations to reach 85% Top-5 accuracy, against 6 iterations with MAM.

show abstract

A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends

Younesi,

Ansari,

Fazli

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Unstructured Weight Pruning in Variability-Aware Memristive Crossbar Neural Networks

Cited by 5 publications

References 30 publications

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

A Multiply-And-Max/min Neuron Paradigm for Aggressively Prunable Deep Neural Networks

A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends

Contact Info

Product

Resources

About