A Novel Deep Learning Model Compression Algorithm

Abstract

In order to solve the problem of large model computing power consumption, this paper proposes a novel model compression algorithm. Firstly, this paper proposes an interpretable weight allocation method for the loss between a student network (a network model with poor performance), a teacher network (a network model with better performance) and real label. Then, different from the previous simple pruning and fine-tuning, this paper performs knowledge distillation on the pruned model, and quantifies the residual weights of the distilled model. The above operations can further reduce the model size and calculation cost while maintaining the model accuracy. The experimental results show that the weight allocation method proposed in this paper can allocate a relatively appropriate weight to the teacher network and real tags. On the cifar-10 dataset, the pruning method combining knowledge distillation and quantization can reduce the memory size of resnet32 network model from 3726 KB to 1842 KB, and the accuracy can be kept at 93.28%, higher than the original model. Compared with similar pruning algorithms, the model accuracy and operation speed are greatly improved.

A Comparative Study of Preprocessing and Model Compression Techniques in Deep Learning for Forest Sound Classification

“…Given the audio signal y(t), STFT representation at time t and frequency t, X(t, f ) can be obtained by Equation (1), where w(t) is the window function and τ is the integration variable. The instantaneous phase can be extracted from the STFT as in Equation (2). Then, the stretched phase is calculated as in Equation ( 3), where a is the time stretch factor.…”

“…Although DL models are complex and resource-intensive, these models can be leveraged to fit into resource-constrained environments by techniques such as model pruning, quantization, and knowledge distillation [ 2 , 3 ]. These techniques, aimed at reducing both the size and computational complexity of DL models, come with a challenge in achieving an optimal trade-off between complexity reduction and sustained performance.…”

The Application of Convolutional Neural Networks in Organ-on-a-Chip Technology: A Review

“…With limited resources and funding, the optimization of CNNs (Habib & Qureshi, 2022) is crucial to their scalability in OoCs. To address the issue of scalability, customized design of models, compression of model designs (Zhao et al, 2022), and iterative improvements to models over time as deep learning progresses are all actions that could be taken in order to increase computational efficiency.…”

Deep Learning Architecture Improvement Based on Dynamic Pruning and Layer Fusion

“…However, the enormous numbers of computations and parameters of CNNs hinder further development. Thus, it is not practical to deploy heavy CNNs on resource-constrained computing devices, such as embedded systems and mobile devices [14][15][16]. To address the problems, substantial research efforts have been devoted to compression techniques: channel pruning [17][18][19][20], low-rank decomposition [21][22][23], and weight quantization [24,25].…”