Discrete Model Compression With Resource Constraint for Deep Neural Networks

Gao, Shangqian; Huang, Feihu; Pei, Jian; Huang, Heng

doi:10.1109/cvpr42600.2020.00197

Cited by 60 publications

(26 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We compare PSS-Net with some compact models including channel pruning and Network Architecture Search (NAS) methods. Channel pruning methods include PFS (Wang et al 2020), MetaPruning , MDP (Liu et al 2020), AMC (He et al 2018), NetAdapt (Yang et al 2018), EagleEye , AutoSlim (Yu and Huang 2019a), DPFPS (Ruan et al 2021), CC (Li et al 2021), DMCP ), CPLI (Guo, Ouyang, and Xu 2020a) and DMC (Gao et al 2020). NAS methods include DARTS (Liu, Simonyan, and Yang 2019), PNAS (Liu et al 2018), NASNet (Zoph et al 2018) and ChamNet (Dai et al 2019).…”

Section: Resultsmentioning

confidence: 99%

Prioritized Subnet Sampling for Resource-Adaptive Supernet Training

chen¹,

Lin²,

Cao³

et al. 2021

Preprint

View full text Add to dashboard Cite

A resource-adaptive supernet adjusts its subnets for inference to fit the dynamically available resources. In this paper, we propose Prioritized Subnet Sampling to train a resourceadaptive supernet, termed PSS-Net. We maintain multiple subnet pools, each of which stores the information of substantial subnets with similar resource consumption. Considering a resource constraint, subnets conditioned on this resource constraint are sampled from a pre-defined subnet structure space and high-quality ones will be inserted into the corresponding subnet pool. Then, the sampling will gradually be prone to sampling subnets from the subnet pools. Moreover, the one with a better performance metric is assigned with higher priority to train our PSS-Net, if sampling is from a subnet pool. At the end of training, our PSS-Net retains the best subnet in each pool to entitle a fast switch of high-quality subnets for inference when the available resources vary. Experiments on ImageNet using MobileNetV1/V2 show that our PSS-Net can well outperform state-of-the-art resource-adaptive supernets. Our project is at https://github.com/chenbong/PSS-Net.

show abstract

Section: Resultsmentioning

confidence: 99%

Prioritized Subnet Sampling for Resource-Adaptive Supernet Training

chen¹,

Lin²,

Cao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Particularly, we found that the implementation of previous pruning algorithms have many notable differences in their retraining step: some employed a small value of learning rate (e.g. 0.001 on ImageNet) to fine-tune the network (Molchanov et al, 2016;Han et al, 2015) for a small number of epochs, e.g., 20 epochs in the work by ; some used a larger value of learning rate (0.01) with much longer retraining budgets, e.g., 60, 100 and 120 epochs respectively on ImageNet (Zhuang et al, 2018;Gao et al, 2020;Li et al, 2020) Figure 1: Learning rate with different schedules on CIFAR when retraining for 72 epochs. In (a), the learning rate is fixed to the last learning rate of original training (i.e.…”

Section: Preliminary and Methodologymentioning

confidence: 99%

Network Pruning That Matters: A Case Study on Retraining Variants

Le¹,

Hua²

2021

Preprint

View full text Add to dashboard Cite

Network pruning is an effective method to reduce the computational expense of over-parameterized neural networks for deployment on low-resource systems. Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the traditional fine-tuning technique in recovering the lost accuracy (Renda et al., 2020), but so far it is unclear what accounts for such performance. In this work, we conduct extensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding. We find that the reason behind the success of learning rate rewinding is the usage of a large learning rate. Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e.g., the 1-cycle learning rate schedule (Smith & Topin, 2019). By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (fine-tuned with the conventional approach). Our results emphasize the cruciality of the learning rate schedule in pruned network retraining -a detail often overlooked by practioners during the implementation of network pruning. INTRODUCTIONTraining neural networks is an everyday task in the era of deep learning and artificial intelligence. Generally speaking, given data availability, large and cumbersome networks are often preferred as they have more capacity to exhibit good data generalization. In the literature, large networks are considered easier to train than small ones (

show abstract

“…  = (0.1, 0.9) 部署两组实验, 模型压缩后对应的精度分别为 70.55%, 70.53%, 显然 1 2 ( , ) [27] 、 Li et al [7] 、 ThiNet [6] 、 Slimming [8] 、 NRE [28] 、 SSS [29] 、 DCP [13] 、 SFP [30] 、 ASFP [31] 、 FPGM [32] 、 AMC [33] 、 COP [34] 、 KSE [35] 、 GAL [15] 、 HRank [36] 、 DHP [37] 、 SFS&DFS [38] 、LFPC [39] 和 DMC [40] . 参考文献…”

Section: 实验设置unclassified