Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey

Abstract

State-of-the-art deep learning models have a parameter count that reaches into the billions. Training, storing and transferring such models is energy and time consuming, thus costly. A big part of these costs is caused by training the network. Model compression lowers storage and transfer costs, and can further make training more efficient by decreasing the number of computations in the forward and/or backward pass. Thus, compressing networks also at training time while maintaining a high performance is an important research topic. This work is a survey on methods which reduce the number of trained weights in deep learning models throughout the training. Most of the introduced methods set network parameters to zero which is called pruning. The presented pruning approaches are categorized into pruning at initialization, lottery tickets and dynamic sparse training. Moreover, we discuss methods that freeze parts of a network at its random initialization. By freezing weights, the number of trainable parameters is shrunken which reduces gradient computations and the dimensionality of the model's optimization space. In this survey we first propose dimensionality reduced training as an underlying mathematical model that covers pruning and freezing during training. Afterwards, we present and discuss different dimensionality reduced training methods.

Investigating the potential of untrained convolutional layers and pruning in computational pathology

“…Over the years, CNN architectures have progressed and developed alternative network designs with the addition of new features and operations to make them more effective; examples include DenseNet 4 ResNet 5 and EfficientNet 6 . Many authors have reviewed or investigated the effectiveness of untrained components in CNN architectures 7,8,9,10,11 . In this work, we investigate how effective untrained convolutional layers are for feature extraction in a computational pathology task using real world data sourced from an available necrosis detection dataset.…”