Pruning filters with L1-norm and capped L1-norm for CNN compression

Kumar, Aakash; Shaikh, Ali Muhammad; Li, Yun; Bilal, Hazrat; Yin, Baocai

doi:10.1007/s10489-020-01894-y

Cited by 125 publications

(15 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inference-time pruning To learn compact structures via training, training based pruning adds various sparsity regularization to the training loss, such as the 1 -norm based [16,24] importance criteria. In addition, Taylor expansion has been adopted in [22,33] as importance criteria to minimize the loss change caused by pruning.…”

Section: Related Workmentioning

confidence: 99%

Coresets based asynchronous network slimming

et al. 2022

View full text Add to dashboard Cite

Pruning is effective to reduce neural networks’ parameters and accelerate inferences, facilitating deep learning in resource-limited scenarios. This paper proposes an asynchronous pruning method for multi-branch networks on the basis of our previous work on channel coresets constructions, to achieve module-level pruning. Firstly, this paper accelerates coreset based pruning by batch sampling with a sampling probability decided on our-designed importance function. Secondly, this paper gives asynchronous pruning solutions with an in-place distillation of feature maps for deployment on multi-branch networks such as ResNet and SqueezeNet. Thirdly, this paper provides an extension to neuron pruning by grouping weights as channels. During tests on sensitivity of different layers to channel pruning, our method outperforms comparison schemes on object detection networks, indicating advantages of data-independent channel selections in maintaining precision. As shown in tests of asynchronous pruning solutions on multi-branch classification networks, our method further decreases FLOPs with a small accuracy decline on ResNet and acquires a small accuracy increment on SqueezeNet. In tests on neuron pruning, our method achieves an accuracy comparable to existing coreset based pruning methods by two solutions of precision recovery.

show abstract

Section: Related Workmentioning

confidence: 99%

Coresets based asynchronous network slimming

et al. 2022

View full text Add to dashboard Cite

show abstract

“…After that, we introduce tactics for feature map selection with filter pruning. Recently, filter pruning has been in the attention of researchers [13] that calculates the importance of the filter by capped L1-norm, scale factors, or Shannon entropy. The L1-norm regularization assists two tasks: 1.…”

Section: Methodsmentioning

confidence: 99%

“…With some convolutional layers, pruning only 10% of information can achieve over 70% reduction in filters. 93.25% 93.30% 1.5 x 10 7 5.4 x 10 6 64.0% Slimming [31] 93.66% 93.80% --88.5% Entropy [43] 93.72% 93.97% 1.5 × 10 7 3.5 × 10 5 76.4% Aketi [19] 93.75% 93.80% --90.5% Kumar [13] 93.77% 93.81%…”

Section: Vgg16 On Cifar10 Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Structure Level Pruning of Efficient Convolutional Neural Networks with Sparse Group LASSO

2022

IJMLC

View full text Add to dashboard Cite

The rapid progress of convolutional neural networks (CNNs) in multiple applications of practical implementation is generally hindered by an upsurge in network size and computational complexity. Currently, engineers focus on reducing these problems through compressing the CNNs by pruning filters and their weights. In this paper, we present a fresh and easy-to-use pruning approach that reduces the model size by eliminating complete filters and filter weights based on the sparse group LASSO (Least Absolute Shrinkage and Selection Operator) method across the convolutional layers. More precisely, it regulates the sparsity at the feature level and the group level. During the process of pruning, the unnecessary filters with their weights eliminate directly without sacrificing accuracy in the test, resulting in much compact and slimmer architectures. We experimentally compute the effectiveness of our methodology with various state-of-art CNN models on various benchmark data sets. Mainly, CIFAR-10 data sets applied on VGG-16 model and reduce the parameters approx. 96.1% and saved approx. 83.55% float-point-operations (FLOPs) without sacrificing accuracy and have obtained development in state-of-art.

show abstract

“…To further suppress the impact of outliers or the noise, capped L1-norm has been proposed as a more robust distance metric and has been implemented in many works, including dictionary learning [13], matrix recovery [26], PCA [33], convolutional neural networks [14], etc. Motivated by the successful application of the capped L1-norm, we apply the capped L1-norm to PTSVM and propose a more robust version of PTSVM, capped L1-norm projection twin support vector machine (CPTSVM).…”

mentioning

confidence: 99%

Robust capped L1-norm projection twin support vector machine

Yang

Wang

2023

JIMO

View full text Add to dashboard Cite

<p style='text-indent:20px;'>Projection twin support vector machine (PTSVM) is an effective tool for classification. However, it is sensitive to outliers or the noise due to the utilization of the squared L2-norm distance. To alleviate the sensitivity to outliers or the noise, we propose a capped L1-norm projection twin support vector machine (CPTSVM), where the L2-norm distance is replaced by the capped L1-norm to confer the robustness to classifiers. CPTSVM is formulated as a pair of non-convex and non-smooth SVM-type problems. To solve these difficult problems, we present an iterative algorithm for CPTSVM as well as its convergence properties. Numerical experiments on artificial and benchmark datasets demonstrate the robustness and feasibility of our proposed method.</p>

show abstract

Pruning filters with L1-norm and capped L1-norm for CNN compression

Cited by 125 publications

References 25 publications

Coresets based asynchronous network slimming

Coresets based asynchronous network slimming

Structure Level Pruning of Efficient Convolutional Neural Networks with Sparse Group LASSO

Robust capped L1-norm projection twin support vector machine

Contact Info

Product

Resources

About