Filter Pruning and Re-Initialization via Latent Space Clustering

Abstract

Filter pruning is prevalent for pruning-based model compression. Most filter pruning methods have two main issues: 1) the pruned network capability depends on that of source pretrained models, and 2) they do not consider that filter weights follow a normal distribution. To address these issues, we propose a new pruning method employing both weight re-initialization and latent space clustering. For latent space clustering, we define filters and their feature maps as vertices and edges to be a graph, transformed into a latent space by graph convolution, alleviating to prune zero-near weight filters only. In addition, a part of filters is re-initialized with a constraint for enhancing filter diversity, and thus the pruned model is less dependent on the source network. This approach provides more robust accuracy even when pruned from the pretrained model with low accuracy. Extensive experimental results show our method decreases 56.6% and 84.6% of FLOPs and parameters of VGG16 with negligible loss of accuracy on CIFAR100, which is the state-of-the art performance. Furthermore, our method presents outperforming or comparable pruning results against state-of-the-art models on multiple datasets.

WLD-Reg: A Data-Dependent Within-Layer Diversity Regularizer

“…Diversity promoting strategies have been widely used in ensemble learning (Li, Yu, and Zhou 2012;Yu, Li, and Zhou 2011), sampling (Bıyık et al 2019;Derezinski, Calandriello, and Valko 2019;Gartrell et al 2019), energybased models (Laakom et al 2021b;Zhao, Mathieu, andLe-Cun 2017), ranking (Gan et al 2020;Yang, Gkatzelis, and Stoyanovich 2019), pruning by reducing redundancy (He et al 2019;Kondo and Yamauchi 2014;Lee et al 2020;Singh et al 2020), and semi-supervised learning (Zbontar et al 2021). In the deep learning context, various approaches have used diversity as a direct regularizer on top of the weight parameters.…”

Reducing Redundancy in the Bottleneck Representation of the Autoencoders

“…In the context of supervised neural networks, it has been shown that reducing the correlation improves generalization [20], [21], [22]. Approaches helping to reduce the redundancy have been successfully applied, e.g., for network pruning [23], [24], [25], [26] and self-supervised learning [27]. In this paper, we propose to model the feature redundancy in the bottleneck representation and minimize it explicitly.…”