Model Compression for Deep Neural Networks: A Survey

Abstract

Currently, with the rapid development of deep learning, deep neural networks (DNNs) have been widely applied in various computer vision tasks. However, in the pursuit of performance, advanced DNN models have become more complex, which has led to a large memory footprint and high computation demands. As a result, the models are difficult to apply in real time. To address these issues, model compression has become a focus of research. Furthermore, model compression techniques play an important role in deploying models on edge devices. This study analyzed various model compression methods to assist researchers in reducing device storage space, speeding up model inference, reducing model complexity and training costs, and improving model deployment. Hence, this paper summarized the state-of-the-art techniques for model compression, including model pruning, parameter quantization, low-rank decomposition, knowledge distillation, and lightweight model design. In addition, this paper discusses research challenges and directions for future work.

Multibit, Lead‐Free Cs2SnI6 Resistive Random Access Memory with Self‐Compliance for Improved Accuracy in Binary Neural Network Application

“…[ 18 ] A primary challenge in contemporary DNN acceleration lies in the extensive data movement between on‐chip processors and off‐chip memory. [ 19 ] In contrast, the binary neural network (BNN) is garnering interest due to its significantly reduced memory requirements (Figure 1a‐i). For a highly integrated BNN hardware system, an RRAM crossbar array (RCA) is particularly suitable, boosting the potential to achieve the most compact design with an area as small as 4F 2 .…”

“…The film has Cs-rich (24.15 ± 0.54)%, Snrich (13.18 ± 0.32)%, and I-deficient (62.67 ± 0.87)%, whereas the compositions of the synthesized particles contrast notably with increased Cs(+3.55%), Sn (+1.38%) (or reduced, I (−4.93%)) atomic % variation. The previous research suggests that process parameters like solvent choice, [9,19,22] the initial concentration of CSI particles in the solvent, and CSI paste-making procedures, including ultrasonication, magnetic stirring, and film-processing techniques, [22] impact the morphology of film and its elemental composition. [9,19,[22][23][24] In this study, the solvent and procedures involved in CSI paste-making and annealing treatments could be responsible for the observed alterations in morphology and chemical composition.…”

“…The previous research suggests that process parameters like solvent choice, [9,19,22] the initial concentration of CSI particles in the solvent, and CSI paste-making procedures, including ultrasonication, magnetic stirring, and film-processing techniques, [22] impact the morphology of film and its elemental composition. [9,19,[22][23][24] In this study, the solvent and procedures involved in CSI paste-making and annealing treatments could be responsible for the observed alterations in morphology and chemical composition. Broad scanning survey XPS spectra of the CSI RS layer confirm the presence of the constituents Cs, Sn, I, and a reference carbon element (Figure S8a,b, Supporting Information).…”

Emotion Recognition and Management in the Tourism Industry During Emergency Events Using Improved Convolutional Neural Network

“…CNN is a type of deep learning model with a core structure that includes convolutional layers for sliding operations with convolutional kernels to extract features from input data, pooling layers to reduce the size of feature maps and computational complexity, activation functions for introducing non-linearity, and fully connected layers for the final layer connection to perform classification or regression [29].…”

Containerization in Edge Intelligence: A Review

“…The third approach is knowledge distillation-training a smaller student model using a larger teacher or multiple models. There are many more approaches, described in greater detail in the following surveys [65,67,68].…”