Improved Bayesian Compression

Federici, Marco Orsini; Ullrich, Karen; Welling, Max

doi:10.48550/arxiv.1711.06494

Cited by 8 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Others, on reducing the cardinality of the networks parameters [27]- [32]. And some have proposed general compression frameworks with the objective to do both, sparsification and cardinality reduction [11], [21], [33]. Our entropy-constrained minimization objective (3) (and its continuous relaxation (6)) can be interpreted as a generalization of those works.…”

Section: Related Workmentioning

confidence: 99%

“…Although minimizing the variational lower bound is also well motivated from a MDL point of view [35], the resulting coding scheme is often impractical for real world scenarios. Therefore, [21], [33], [36] focused on designing suitable priors and posteriors that allow to apply practical coding schemes on the weight parameters after the variational lower bound has been minimized. This includes a final step where a lossless entropy coder is applied to the network's parameter as proposed by [11].…”

Section: Related Workmentioning

confidence: 99%

“…Afterwards, in the evaluation stage, the network weights were quantized using the values of the vector Ω l , available in the l-th layer. We set their respective initial cardinalities K as follows: (i) LeNet-300-100, K = [3,3,33], (ii) LeNet-5, K = [5,5,33], and (iii) VGG-Cifar-10, K = [33,17,15,11,7,7,7,5,5,5,3,3,3,17,33]. The quantized network performance was monitored with the classification error, compression ratio, and sparsity (calculated on the weights).…”

Section: A Experimental Setupmentioning

confidence: 99%

See 2 more Smart Citations

Entropy-Constrained Training of Deep Neural Networks

Wiedemann

Müller

Samek

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

We propose a general framework for neural network compression that is motivated by the Minimum Description Length (MDL) principle. For that we first derive an expression for the entropy of a neural network, which measures its complexity explicitly in terms of its bit-size. Then, we formalize the problem of neural network compression as an entropyconstrained optimization objective. This objective generalizes many of the compression techniques proposed in the literature, in that pruning or reducing the cardinality of the weight elements of the network can be seen special cases of entropy-minimization techniques. Furthermore, we derive a continuous relaxation of the objective, which allows us to minimize it using gradient based optimization techniques. Finally, we show that we can reach stateof-the-art compression results on different network architectures and data sets, e.g. achieving x71 compression gains on a VGG-like architecture. W j 8 z P K V s Q k f c t z S h M T d B P o 8 8 I 2 d W G Z J I a f s S J H P 1 9 0 Z O Y 2 O m c W g n i 4 h m 2 S v E / z w / w + g 6 y E W S Z s g T t v g o y i R B R Y r 7 y V B o z l B O L a F M C 5 u V s D H V l K F t q W Z L 8 J Z P X i X d i 6 b n N r 3 7 y 0 b r p q y j C i d w C u f g w R W 0 4 A 7 a 0 A E W j 8 z P K V s Q k f c t z S h M T d B P o 8 8 I 2 d W G Z J I a f s S J H P 1 9 0 Z O Y 2 O m c W g n i 4 h m 2 S v E / z w / w + g 6 y E W S Z s g T t v g o y i R B R Y r 7 y V B o z l B O L a F M C 5 u V s D H V l K F t q W Z L 8 J Z P X i X d i 6 b n N r 3 7 y 0 b r p q y j C i d w C u f g w R W 0 4 A 7 a 0 A E W j 8 z P K V s Q k f c t z S h M T d B P o 8 8 I 2 d W G Z J I a f s S J H P 1 9 0 Z O Y 2 O m c W g n i 4 h m 2 S v E / z w / w + g 6 y E W S Z s g T t v g o y i R B R Y r 7 y V B o z l B O L a F M C 5 u V s D H V l K F t q W Z L 8 J Z P X i X d i 6 b n N r 3 7 y 0 b r p q y j C i d w C u f g w R W 0 4 A 7 a 0 A E W j 8 z P K V s Q k f c t z S h M T d B P o 8 8 I 2 d W G Z J I a f s S J H P 1 9 0 Z O Y 2 O m c W g n i 4 h m 2 S v E / z w / w + g 6 y E W S Z s g T t v g o y i R B R Y r 7 y V B o z l B O L a F M C 5 u V s D H V l K F t q W Z L 8 J Z P X i X d i 6 b n N r 3 7 y 0 b r p q y j C i d w C u f g w R W 0 4 A 7 a 0 A E W j 8 z P K V s Q k f c t z S h M T d B P o 8 8 I 2 d W G Z J I a f s S J H P 1 9 0 Z O Y 2 O m c W g n i 4 h m 2 S v E / z w / w + g 6 y E W S Z s g T t v g o y i R B R Y r 7 y V B o z l B O L a F M C 5 u V s D H V l K F t q W Z L 8 J Z P X i X d i 6 b n N r 3 7 y 0 b r p q y j C i d w C u f g w R W 0 4 A 7 a 0 A E W j 8 z P K V s Q k f c t z S h M T d B P o 8 8 I 2 d W G Z J I a f s S J H P 1 9 0 Z O Y 2 O m c W g n i 4 h m 2 S v E / z w / w + g 6 y E W S Z s g T t v g o y i R B R Y r 7 y V B o z l B O L a F M C 5 u V s D H V l K F t q W Z L 8 J Z P X i X d i 6 b n N r 3 7 y 0 b r p q y j C i d w C u f g w R W 0 4 A 7 a 0 A E W j 8 z P K V s Q k f c t z S h M T d B P o 8 8 I 2 d W G Z J I a f s S J H P 1 9 0 Z O Y 2 O m c W g n i 4 h m 2 S v E / z w / w + g 6 y E W S Z s g T t v g o y i R B R Y r 7 y V B o z l B O L a F M C 5 u V s D H ...

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: A Experimental Setupmentioning

confidence: 99%

See 1 more Smart Citation

Entropy-Constrained Training of Deep Neural Networks

Wiedemann

Müller

Samek

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…Han et al (2015a) assembles pruning, quantization and Huffman coding to achieve better compression rate. Bayesian methods Ullrich et al (2017); Louizos et al (2017); Federici et al (2017) are also used to retrain the model such that the model has more space to be compressed. He et al (2018) uses reinforcement learning to design a compression algorithm.…”

Section: Related Work On Model Compressionmentioning

confidence: 99%

“…The other category directly compresses a given large neural network using pruning, quantization, and matrix factorization, including LeCun et al (1990); Hassibi and Stork (1993); Han et al (2015b,a); Cheng et al (2015). There are also advanced methods to train the neural network using Bayesian methods to help pruning or quantization at a later stage, such as Ullrich et al (2017); Louizos et al (2017); Federici et al (2017).…”

Section: Introductionmentioning

confidence: 99%

Rate Distortion For Model Compression: From Theory To Practice

Gao¹,

Liu²,

Wang³

et al. 2018

Preprint

View full text Add to dashboard Cite

The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main technical building blocks are parameter pruning, parameter sharing (quantization), and low-rank factorization. In this paper, we propose principled approaches to improve upon the common heuristics used in those building blocks, namely pruning and quantization.We first study the fundamental limit for model compression via the rate distortion theory. We bring the rate distortion function from data compression to model compression to quantify this fundamental limit. We prove a lower bound for the rate distortion function and prove its achievability for linear models. Although this achievable compression scheme is intractable in practice, this analysis motivates a novel model compression framework. This framework provides a new objective function in model compression, which can be applied together with other classes of model compressor such as pruning or quantization. Theoretically, we prove that the proposed scheme is optimal for compressing one-hidden-layer ReLU neural networks. Empirically, we show that the proposed scheme improves upon the baseline in the compression-accuracy tradeoff.

show abstract

Group Pruning Using a Bounded- $$\ell _p$$ ℓ p Norm for Group Gating and Regularization

Mummadi

Genewein

Zhang

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Deep neural networks achieve state-of-the-art results on several tasks while increasing in complexity. It has been shown that neural networks can be pruned during training by imposing sparsity inducing regularizers. In this paper, we investigate two techniques for group-wise pruning during training in order to improve network efficiency. We propose a gating factor after every convolutional layer to induce channel level sparsity, encouraging insignificant channels to become exactly zero. Further, we introduce and analyse a bounded variant of the 1 regularizer, which interpolates between 1 and 0-norms to retain performance of the network at higher pruning rates. To underline effectiveness of the proposed methods, we show that the number of parameters of ResNet-164, DenseNet-40 and MobileNetV2 can be reduced down by 30%, 69%, and 75% on CIFAR100 respectively without a significant drop in accuracy. We achieve state-of-the-art pruning results for ResNet-50 with higher accuracy on ImageNet. Furthermore, we show that the light weight Mo-bileNetV2 can further be compressed on ImageNet without a significant drop in performance. IntroductionModern deep neural networks are notoriously known for requiring large computational resources, which becomes particularly problematic in resource-constrained domains, such as in automotive, mobile or embedded applications. Neural network compression methods aim at reducing the computational footprint of a neural network while preserving task performance (e.g. classification accuracy) [4,34]. One family of such methods, Network pruning, operates by removing unnecessary weights or even whole neurons or convolutional featuremaps ("channels") during or after training, thus reducing computational resources needed at test time or deployment. A simple relevance-criterion for pruning weights is weight-magnitude: "small" weights contribute relatively little to the overall computation (dot-products and convolutions) and can thus be removed.However, weight-pruning leads to unstructured sparsity in weight matrices and filters. While alleviating storage demands, it is non-trivial to exploit

show abstract

Improved Bayesian Compression

Cited by 8 publications

References 0 publications

Entropy-Constrained Training of Deep Neural Networks

Entropy-Constrained Training of Deep Neural Networks

Rate Distortion For Model Compression: From Theory To Practice

Group Pruning Using a Bounded- $$\ell _p$$ ℓ p Norm for Group Gating and Regularization

Contact Info

Product

Resources

About