Exploring Gradient Flow Based Saliency for DNN Model Compression

Abstract

Model pruning aims to reduce the deep neural network (DNN) model size or computational overhead. Traditional model pruning methods such as ℓ 1 pruning that evaluates the channel significance for DNN pay too much attention to the local analysis of each channel and make use of the magnitude of the entire feature while ignoring its relevance to the batch normalization (BN) and ReLU layer after each convolutional operation. To overcome these problems, we propose a new model pruning method from a new perspective of gradient flow in this paper. Specifically, we first theoretically analyze the channel's influence based on Taylor expansion by integrating the effects of BN layer and ReLU activation function. Then, the incorporation of the first-order Talyor polynomial of the scaling parameter and the shifting parameter in the BN layer is suggested to effectively indicate the significance of a channel in a DNN. Comprehensive experiments on both image classification and image denoising tasks demonstrate the superiority of the proposed novel theory and scheme. Code is available at https://github.com/CityU-AIM-Group/GFBS. CCS CONCEPTS• Computing methodologies → Machine learning; Model compression; • Computer systems organization → Neural networks.

Efficient tensor decomposition-based filter pruning

“…Pruning MobileNet-v2 presents a significant challenge due to its exceedingly low computational cost. However, CORING exhibits superior performance compared to other candidate methods [45,66], achieving a top-1 accuracy of 94.81% while pruning 42% of the network's MACs, as illustrated in Table 5. Even when compressing more than 60% of the network, the accuracy is not reduced, which suggests that CORING can be applied to optimize the hand-crafted design networks.…”

“…4, illustrating the effectiveness of CORING. [31] LAASP (2023) [16] HRel (2022) [56] DECORE (2022) [1] EZCrop (2022) [40] FPFS (2022) [79] WSP (2021) [18] CHIP (2021) [63] GFBS (2021) [45] HRank (2020) [38] Figure 4: Pruning methods for VGG-16 baseline on CIFAR-10.…”

“…In the scenario involving significant compression, with approximately 70% compression, CORING outperforms a recent SOTA method [34] in all aspects. FPAC [76] 93.66 0.39M(61.9) 113.08M(59.9) HRank-2 [38] 93.68 0.48M(53.8) 110.15M(61.0) EZCrop [40] 93.76 0.39M(61.9) 113.08M(59.9) DECORE-70 [1] 94.04 0.37M(65.0) 128.13M(54.7) CORING (Ours) 94.30 0.45M(57.3) 134.86M(53.5) QSFM-PSNR [73] 92.06 1.67M(25.4) 57.27M(39.4) DMC [15] 94.49 N/A 56.72M(40.0) SCOP [66] 94.24 1.43M(36.1) 56.44M(40.3) GFBS [45] 94.25 N/A 54.83M(42.0) CORING (Ours) 94.44 0.77M(65.6) 38.00M(60.0) DenseNet-40. Managing DenseNet architecture can be challenging because removing a single channel from the architecture requires removing that channel from all subsequent layers [1].…”

“…Comparison. CORING is compared with 44 SOTAs in the fields of structured pruning [1,3,5,6,7,11,12,15,16,18,19,23,26,29,30,31,32,34,35,36,37,38,39,40,41,43,45,46,47,48,49,56,58,63,64,66,68,73,76,78,79,80,83,85]. For a fair comparison, all available baseline models are identical.…”