VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks

Abstract

Improving weight sparsity is a common strategy for producing light-weight deep neural networks. However, pruning models with residual learning is more challenging. In this paper, we introduce Variance-Aware Cross-Layer (VACL), a novel approach to address this problem. VACL consists of two parts, a Cross-Layer grouping and a Variance Aware regularization. In Cross-Layer grouping the i th filters of layers connected by skip-connections are grouped into one regularization group. Then, the Variance-Aware regularization term takes into account both the first and second-order statistics of the connected layers to constrain the variance within a group. Our approach can effectively improve the structural sparsity of residual models. For CIFAR10, the proposed method reduces a ResNet model by up to 79.5% with no accuracy drop, and reduces a ResNeXt model by up to 82% with < 1% accuracy drop. For ImageNet, it yields a pruned ratio of up to 63.3% with < 1% top-5 accuracy drop. Our experimental results show that the proposed approach significantly outperforms other state-of-the-art methods in terms of overall model size and accuracy.

Acceleration-Aware Fine-Grained Channel Pruning for Deep Neural Networks via Residual Gating

“…Lemaire et al [34] use a mixed block connectivity to avoid redundant computation, which can be treated as a subset of our method. Recently, a group pruning strategy [21], [22], [31], [35] is proposed to assign those layers into the same group, thus filters in the same group can be pruned simultaneously. Unfortunately, pruning in a group strategy requires that all connections of one pruned filter should be removed simultaneously, of which the constraint is so strong that limits the pruning performance at especially high pruning ratios.…”

“…The biggest difference between our pruned residual neural network and existing methods is that we consider the problem of pruning from the perspective of gating feature maps, the gate function is induced on each channel while skip connections are always retained. One simple strategy is to prune each channel independently without any constraint among channels or layers, thus a better trade-off between compression ratio and performance should be achieved due to the larger search space of channels than strategies such as group pruning [21], [22] or skipping [19], [20]. However, too much freedom leads to irregular distributions of pruned channels between residual blocks, and during experiments we found the advantage of such strategy over group pruning or skipping is not significant as expected especially for more complex models and tasks.…”

HALP: Hardware-Aware Latency Pruning

“…Since MobileNet has group convolutional layers to speedup the inference, we take the group convolutional layer with its preceding connected convolutional layer together as coupled crosslayers Gao et al (2019) to make sure the input channel number and output channel number of the group convolution remain the same. All the 27 convolutional layers can be divided into 14 coupled layers.…”

When to Prune? A Policy towards Early Structural Pruning

“…Prune while training methods rest in the middle by finding a trade-off between training efficiency and final accuracy. Literature falls under two streams towards this task: a) regularization-based methods that encourage sparsity during training [3,13,28], and b) sub-ticket selection methods via saliency that discard redundancy [2,14,17]. Our work belongs to the latter given its efficacy to quickly enforce a pruning ratio and ease-of-control during training.…”