Graph pruning for model compression

Abstract

Previous AutoML pruning works utilized individual layer features to automatically prune filters. We analyze the correlation for two layers from different blocks which have a short-cut structure. It is found that, in one block, the deeper layer has many redundant filters which can be represented by filters in the former layer so that it is necessary to take information from other layers into consideration in pruning. In this paper, a graph pruning approach is proposed, which views any deep model as a topology graph. Graph Prun-ingNet based on the graph convolution network is designed to automatically extract neighboring information for each node. To extract features from various topologies, Graph PruningNet is connected with Pruned Network by an individual fully connection layer for each node and jointly trained on a training dataset from scratch. Thus, we can obtain reasonable weights for any size of sub-network. We then search the best configuration of the Pruned Network by reinforcement learning. Different from previous work, we take the node features from well-trained Graph Prun-ingNet, instead of the hand-craft features, as the states in reinforcement learning. Compared with other AutoML pruning works, our method has achieved the state-of-the-art under same conditions on ImageNet-2012. The code will be released on GitHub.

BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

“…We refer to [54] for a more comprehensive NAS review. Block-wise weight-sharing NAS [36,46,83,84] approaches factorize the supernet into independently optimized blocks and thus reduce the weight-sharing space, resolving the issue of inaccurate architecture ratings caused by weightsharing. DNA [36] first introduced the block-wisely supervised architecture rating scheme with knowledge distillation.…”

“…Based on this scheme, DONNA [46] further propose to predict an architecture rating using a linear combination of its blockwise ratings rather than a simplistic sum. SP [83] were the first to apply this scheme to network pruning. However, all of the aforementioned methods rely on a supervised distillation scheme, which inevitably introduces architectural bias from the teacher.…”

“…In this section, we first briefly introduce the dilemma of NAS and its block-wise solutions [36,46,83,84], then present our proposed BossNAS in detail, along with its two key elements: i) unsupervised supernet training phase with ensemble bootstrapping; ii) unsupervised architecture rating and searching phase towards architecture population center. Notations.…”

“…total number of weight-sharing architectures) can effectively improve the accuracy of architecture rating. In practice, block-wise solutions [36,46,83,84] find a way out of this dilemma of NAS by block-wisely factorizing the search space in the depth dimension, thus reducing the weight-sharing space while maintaining the original size of the search space. Given a supernet consisting of |k| blocks…”

Model Compression

“…H. Zhu and Suyog Gupta showed comparison between the performance of large-sparse models and small dense model on a large variety of datasets and shows that large sparse model outperforms the former. Zhang, et al in their paper Graph Pruning for Model Compression discuss in filter pruning used in conjunction with graph convolution [3]. Filter pruning is a method in which selected filters are removed and a narrower model is rebuilt.…”