Pruning the Convolution Neural Network (SqueezeNet) based on L&lt;inf&gt;2&lt;/inf&gt; Normalization of Activation Maps

Gaikwad, Akash; El-Sharkawy, Mohamed

doi:10.1109/ccwc.2019.8666597

Cited by 3 publications

(5 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Input Hence, the saliency of a channel can be regarded as a function of outputs [22], [36]- [38], rather than parameters, i.e. in Equation 1X can be either the weights, l W i , or the output feature map, l A i .…”

Section: Base Pointwise Metric Reduction Scalingmentioning

confidence: 99%

“…Any suitable vector norm could be used as a reduction, with the L2-norm being a popular reduction method in the literature [36], [41].…”

Section: Dimensionality Reduction (Choice Of R)mentioning

confidence: 99%

“…Data driven approaches which consider gradients or output points in the domain X often rely on a set of inputs to produce these values. Since each input in the set will result in a potentially different set of saliency values, the computation of the saliency over a minibatch is typically done by combining the element-wise saliency values using a simple average across the minibatch [22], [23], [36], [40].…”

Section: E Minibatchesmentioning

confidence: 99%

“…In some cases, a square root is applied to the resulting saliency metric [36]. This additional computation does not modify the ranking of the channels and can thus be omitted for algorithms only concerned with channel ranking.…”

Section: E Minibatchesmentioning

confidence: 99%

“…Output values across inferences on multiple inputs can be gathered, and their results summarized using statistical measures. Saliency metrics use the sum [37], mean and variance [38], or L2-norm [36] of feature maps to identify low saliency channels.…”

Section: Weight and Input Images Based Saliency Metricsmentioning

confidence: 99%

See 4 more Smart Citations

Taxonomy of Saliency Metrics for Channel Pruning

Persand¹,

Anderson

Gregg

2021

IEEE Access

View full text Add to dashboard Cite

Pruning unimportant parameters can allow deep neural networks (DNNs) to reduce their heavy computation and memory requirements. A saliency metric estimates which parameters can be safely pruned with little impact on the classification performance of the DNN. Many saliency metrics have been proposed, each within the context of a wider pruning algorithm. The result is that it is difficult to separate the effectiveness of the saliency metric from the wider pruning algorithm that surrounds it. Similar-looking saliency metrics can yield very different results because of apparently minor design choices. We propose a novel taxonomy of saliency metrics based on four mostly-orthogonal principal components. We show that a broad range of metrics from the pruning literature can be grouped according to these components. Our taxonomy serves as a guide to prior work, and allows us to construct new saliency metrics by exploring novel combinations of our taxonomic components. We perform the first in-depth experimental investigation of more than 300 saliency metrics made up of existing techniques and new combinations of components. Our results provide decisive answers to open research questions. In particular, we demonstrate the importance of reduction and scaling when pruning groups of weights. We also propose a novel scaling method based on the number of weights transitively removed. We find that some of our constructed metrics can outperform the best existing state-of-the-art metrics for convolutional neural network channel pruning. We find further that our novel scaling method improves existing saliency metrics.INDEX TERMS machine learning, convolution neural networks, pruning, saliency metric, model compression VOLUME x, 20xx

show abstract

Section: Base Pointwise Metric Reduction Scalingmentioning

confidence: 99%

“…Any suitable vector norm could be used as a reduction, with the L2-norm being a popular reduction method in the literature [36], [41].…”

Section: Dimensionality Reduction (Choice Of R)mentioning

confidence: 99%

Section: E Minibatchesmentioning

confidence: 99%

Section: E Minibatchesmentioning

confidence: 99%

Section: Weight and Input Images Based Saliency Metricsmentioning

confidence: 99%

See 3 more Smart Citations

Taxonomy of Saliency Metrics for Channel Pruning

Persand¹,

Anderson

Gregg

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

Survey: Exploiting Data Redundancy for Optimization of Deep Learning

Chen

Niu²,

Ren

et al. 2023

ACM Comput. Surv.

View full text Add to dashboard Cite

Data redundancy is ubiquitous in the inputs and intermediate results of Deep Neural Networks (DNN). It offers many significant opportunities for improving DNN performance and efficiency and has been explored in a large body of work. These studies have scattered in many venues across several years. The targets they focus on range from images to videos and texts, and the techniques they use to detect and exploit data redundancy also vary in many aspects. There is not yet a systematic examination and summary of the many efforts, making it difficult for researchers to get a comprehensive view of the prior work, the state of the art, differences and shared principles, and the areas and directions yet to explore. This article tries to fill the void. It surveys hundreds of recent papers on the topic, introduces a novel taxonomy to put the various techniques into a single categorization framework, offers a comprehensive description of the main methods used for exploiting data redundancy in improving multiple kinds of DNNs on data, and points out a set of research opportunities for future to explore.

show abstract

Indoor Human Motion Recognition Method Based on Kernel-Distance Doppler Velocity Estimation and Lightweight Network

Gao

Yang

et al. 2022

2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)

View full text Add to dashboard Cite

Pruning the Convolution Neural Network (SqueezeNet) based on L<inf>2</inf> Normalization of Activation Maps

Cited by 3 publications

References 4 publications

Taxonomy of Saliency Metrics for Channel Pruning

Taxonomy of Saliency Metrics for Channel Pruning

Survey: Exploiting Data Redundancy for Optimization of Deep Learning

Indoor Human Motion Recognition Method Based on Kernel-Distance Doppler Velocity Estimation and Lightweight Network

Contact Info

Product

Resources

About