Towards Optimal Structured CNN Pruning via Generative Adversarial Learning

Abstract

Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an endto-end manner. To accomplish this, we first introduce a soft mask to scale the output of these structures by defining a new objective function with sparsity regularization to align the output of baseline and network with this mask. We then effectively solve the optimization problem by generative adversarial learning (GAL), which learns a sparse soft mask in a label-free and an end-to-end manner. By forcing more scaling factors in the soft mask to zero, the fast iterative shrinkage-thresholding algorithm (FISTA) can be leveraged to fast and reliably remove the corresponding structures. Extensive experiments demonstrate the effectiveness of GAL on different datasets, including MNIST, CIFAR-10 and Im-ageNet ILSVRC 2012. For example, on ImageNet ILSVRC 2012, the pruned ResNet-50 achieves 10.88% Top-5 error and results in a factor of 3.7× speedup. This significantly outperforms state-of-the-art methods.

Deep-Learning Steganalysis for Removing Document Images on the Basis of Geometric Median Pruning

“…Liu et al [37] used scaling factor γ in the normalization layer to impose sparseness constraints, measure the importance of channels during the training process, filter out channels with low scores and prune layer-by-layer. Huang et al [38] and Lin et al [39] proposed a sparse regularization mask method based on channel pruning; the mask is optimized via data-driven selection or generative adversarial learning. Zhao et al [40] further developed the norm-based importance estimation by taking the dependency between the adjacent layers into consideration and propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity.…”

Filter Pruning and Re-Initialization via Latent Space Clustering

“…However, this assumption is not always satisfied. New approaches also appear based on neural architecture search (NAS) [36], [37] and generative models [38], which require too heavy costs to be useful in practical applications.…”

“…So, when applying a general pruning technique to VGG16, the performance degradation is negligible in general. Fine-tuning may result in a higher performance of the pruned network Li et al [20] VID [34] GAL [38] than a given pretrained network. In fact, we can observe such a phenomenon in GAL [38] and GBN [31].…”

“…Fine-tuning may result in a higher performance of the pruned network Li et al [20] VID [34] GAL [38] than a given pretrained network. In fact, we can observe such a phenomenon in GAL [38] and GBN [31]. It is worth noting that our method not only improves accuracy but also achieves SOTA performance, as shown in Table 2.…”

Adaptive Pruning of Transfer Learned Deep Convolutional Neural Network for Classification of Cervical Pap Smear Images

“…These methods can minimize the local reconstruction error, but may increase the global reconstruction error. In addition, generative adversarial learning [41] and meta-learning [42] have also been applied to improve the pruning algorithm and reduce computational complexity. However, how to adaptively and efficiently select important filters is still a challenge.…”