When to Prune? A Policy towards Early Structural Pruning

Abstract

Data often arrives in sequence over time in real-world deep learning applications such as autonomous driving.When new training data is available, training the model from scratch undermines the benefit of leveraging the learned knowledge, leading to significant training costs. Warm-starting from a previously trained checkpoint is the most intuitive way to retain knowledge and advance learning. However, existing literature suggests that this warmstarting degrades generalization. In this paper, we advocate for warm-starting but stepping out of the previous converging point, thus allowing a better adaptation to new data without compromising previous knowledge. We propose Knowledge Consolidation and Acquisition (CKCA), a continuous model improvement algorithm with two novel components. First, a novel feature regularization (FeatReg) to retain and refine knowledge from existing checkpoints; Second, we propose adaptive knowledge distillation (AdaKD), a novel approach to forget mitigation and knowledge transfer. We tested our method on ImageNet using multiple splits of the training data. Our approach achieves up to 8.39% higher top1 accuracy than the vanilla warm-starting and consistently outperforms the prior art with a large margin.

Dnnshifter: An Efficient Dnn Pruning System for Edge Computing

“…DNNShifter is primarily limited by the high computation cost of training sparse models. There is potential for structured pruning to be conducted at the initialisation of the model (before training) with minimal accuracy loss [49,50]. This will be explored in the future.…”

Lossless Reconstruction of Convolutional Neural Network for Channel-Based Network Pruning

“…Li et al [ 41 ] determined channel configuration using a random search. Shen et al [ 42 ] pruned channels globally based on magnitude and gradient criteria. Unlike pruning-only methods, Hou et al [ 43 ] proposed a pruning-and-regrowing method to avoid removing important channels.…”

Research on Intelligent Management of Laboratory Information Technology

“…Provided there are N tokens of d dimension corresponding with the image patches, the self-attention to correlate every couple from the permutation of the N tokens will result in O(N 2 d) complexity in a simple updating round. For deploying Transformer on edge devices, a variety of simplified models have been proposed, aiming to reduce parameters and operations, for example, parameter pruning [12,28], low-rank factorization [38], and knowledge distillation [24,35]. Yet, these strategies for acceleration are limited in that they still rely on CNN, which deviates from the original design of Transformer, that is, facilitating deep learning with a new working mechanism other than CNN.…”