Is Pruning Compression?: Investigating Pruning Via Network Layer Similarity

Blakeney, Cody; Yan, Yan; Zong, Ziliang

doi:10.1109/wacv45572.2020.9093318

Cited by 8 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On another note, pruning is argued to be important since one cannot achieve the same adversarial robustness or performance by training from scratch [69]. In addition, a recent study [7] confirms the observations of Liu et al [45] and studies the differences between the representations of the pruned nets and vanilla nets.…”

Section: Related Worksupporting

confidence: 68%

“…In the literature, the de facto standard of assessing neuron importance is to check the average absolute value of its weights which is an acceptable proposition for fully connected neural nets [23,22,66,19,41,64,67,63,69,7]. However, for visual recognition, convolutional neural nets (CNNs) [13,37] have already surpassed the fully connected nets in terms of both accuracy and efficiency.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Deeper Look into Convolutions via Pruning

Cugu¹,

Akbaş²

2021

Preprint

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are able to attain better visual recognition performance than fully connected neural networks despite having much less parameters due to their parameter sharing principle. Hence, modern architectures are designed to contain a very small number of fully-connected layers, often at the end, after multiple layers of convolutions. It is interesting to observe that we can replace large fully-connected layers with relatively small groups of tiny matrices applied on the entire image. Moreover, although this strategy already reduces the number of parameters, most of the convolutions can be eliminated as well, without suffering any loss in recognition performance. However, there is no solid recipe to detect this hidden subset of convolutional neurons that is responsible for the majority of the recognition work. Hence, in this work, we use the matrix characteristics based on eigenvalues in addition to the classical weight-based importance assignment approach for pruning to shed light on the internal mechanisms of a widely used family of CNNs, namely residual neural networks (ResNets), for the image classification problem using CIFAR-10, CIFAR-100 and Tiny ImageNet datasets. The codes are available at https://github.com/cuguilke/psykedelic.

show abstract

Section: Related Worksupporting

confidence: 68%

Section: Introductionmentioning

confidence: 99%

A Deeper Look into Convolutions via Pruning

Cugu¹,

Akbaş²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In mathematics, the rank of a smooth function measures the volume of independent information captured by the function [21]. Deep neural networks are highly smooth functions, thus the rank of a network has long been an essential concept in machine learning that underlies many tasks such as information compression [48,56,36,54,49], network pruning [32,55,5,25,9], data mining [6,24,10,57,18,29], computer vision [59,58,31,27,29,60], and natural language processing [8,28,7,11]. Numerous methods are either designed to utilize the mathematical property of network ranks, or are derived from an assumption that low-rank structures are to be preferred.…”

Section: Introductionmentioning

confidence: 99%

Rank Diminishing in Deep Neural Networks

Feng¹,

Zheng²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The rank of neural networks measures information flowing across layers. It is an instance of a key structural condition that applies across broad domains of machine learning. In particular, the assumption of low-rank feature representations leads to algorithmic developments in many architectures. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear. To fill this gap, we perform a rigorous study on the behavior of network rank, focusing particularly on the notion of rank deficiency. We theoretically establish a universal monotonic decreasing property of network rank from the basic rules of differential and algebraic composition, and uncover rank deficiency of network blocks and deep function coupling. By virtue of our numerical tools, we provide the first empirical analysis of the per-layer behavior of network rank in practical settings, i.e., ResNets, deep MLPs, and Transformers on ImageNet. These empirical results are in direct accord with our theory. Furthermore, we reveal a novel phenomenon of independence deficit caused by the rank deficiency of deep networks, where classification confidence of a given category can be linearly decided by the confidence of a handful of other categories. The theoretical results of this work, together with the empirical findings, may advance understanding of the inherent principles of deep neural networks.Preprint. Under review.

show abstract

“…Previous studies [3,17] have shown that pruned neural networks evolve to substantially different representations while striving to preserve overall accuracy. In Section 3, we have demonstrated that knowledge distillation can effectively mitigate both pruning and data induced bias in compressed networks.…”

Section: Explaining Model Bias Using Model Similaritymentioning

confidence: 99%

“…Movva and Zhao [17] investigated the impact of pruning on layer similarities of NLP models using LinearCKA [12]. Ansuini et al and Blakeney et al also investigated how pruning can change representations using similarity based measures [2,3]. Unfortunately, very few work have studied how pruning can induce bias and how to evaluate and mitigate the pruning induced bias.…”

Section: Related Workmentioning

confidence: 99%

Simon Says: Evaluating and Mitigating Bias in Pruned Neural Networks with Knowledge Distillation

Blakeney¹,

Huish²,

Yan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years the ubiquitous deployment of AI has posed great concerns in regards to algorithmic bias, discrimination, and fairness. Compared to traditional forms of bias or discrimination caused by humans, algorithmic bias generated by AI is more abstract and unintuitive therefore more difficult to explain and mitigate. A clear gap exists in the current literature on evaluating and mitigating bias in pruned neural networks. In this work, we strive to tackle the challenging issues of evaluating, mitigating, and explaining induced bias in pruned neural networks. Our paper makes three contributions. First, we propose two simple yet effective metrics, Combined Error Variance (CEV) and Symmetric Distance Error (SDE), to quantitatively evaluate the induced bias prevention quality of pruned models. Second, we demonstrate that knowledge distillation can mitigate induced bias in pruned neural networks, even with unbalanced datasets. Third, we reveal that model similarity has strong correlations with pruning induced bias, which provides a powerful method to explain why bias occurs in pruned neural networks. Our code is available at https://github.com/codestar12/pruning-distilation-bias Preprint. Under review.

show abstract

Is Pruning Compression?: Investigating Pruning Via Network Layer Similarity

Cited by 8 publications

References 7 publications

A Deeper Look into Convolutions via Pruning

A Deeper Look into Convolutions via Pruning

Rank Diminishing in Deep Neural Networks

Simon Says: Evaluating and Mitigating Bias in Pruned Neural Networks with Knowledge Distillation

Contact Info

Product

Resources

About