Compressing by Learning in a Low-Rank and Sparse Decomposition Form

Guo, Kailing; Xie, Xiaona; Xu, Xiangmin; Xing, Xiaofen

doi:10.1109/access.2019.2947846

Cited by 9 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In [18], authors speed up convolutional layers by using depth-wise separable convolution and show a way to find the optimal channel configuration in groups to prevent a quality decrease. Modern methods also combine different approaches, for example, lowrank approximations and sparse filters, and reduce the number of parameters of modern architectures by about 30%, leaving accuracy almost at the same level [19].…”

Section: Related Workmentioning

confidence: 99%

ResNet-like Architecture with Low Hardware Requirements

Limonova¹,

Alfonso²,

Nikolaev

et al. 2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

One of the most computationally intensive parts in modern recognition systems is an inference of deep neural networks that are used for image classification, segmentation, enhancement, and recognition. The growing popularity of edge computing makes us look for ways to reduce its time for mobile and embedded devices. One way to decrease the neural network inference time is to modify a neuron model to make it more efficient for computations on a specific device. The example of such a model is a bipolar morphological neuron model. The bipolar morphological neuron is based on the idea of replacing multiplication with addition and maximum operations. This model has been demonstrated for simple image classification with LeNet-like architectures [1]. In the paper, we introduce a bipolar morphological ResNet (BM-ResNet) model obtained from a much more complex ResNet architecture by converting its layers to bipolar morphological ones. We apply BM-ResNet to image classification on MNIST and CIFAR-10 datasets with only a moderate accuracy decrease from 99.3% to 99.1% and from 85.3% to 85.1%. We also estimate the computational complexity of the resulting model. We show that for the majority of ResNet layers, the considered model requires 2.1-2.9 times fewer logic gates for implementation and 15-30% lower latency.

show abstract

Section: Related Workmentioning

confidence: 99%

ResNet-like Architecture with Low Hardware Requirements

Limonova¹,

Alfonso²,

Nikolaev

et al. 2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

show abstract

“…Sparse feature learning [14]- [19], L+S regularization of network weights [20] (U-Net trained with L+S output loss)…”

Section: Neural-model-based Mathematical-model-basedmentioning

confidence: 99%

“…There is plenty of room to incorporate sparsity-inducing priors into training as knowledge injection. They have explicitly been utilized for sparsification of hidden unit outputs in autoencoder-based sparse feature learning [14]- [19] as well as for low-rank and/or sparse regularization of network weights [5], [20]. The nuclear norm has not been used as a loss function yet, even though it is backpropable via automatic differentiation of the singular value decomposition [21].…”

Section: None (But Crucial To Prevent Catastrophic Forgetting)mentioning

confidence: 99%

Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors

Sakai

2021

Preprint

View full text Add to dashboard Cite

What if deep neural networks can learn from sparsity-inducing priors? When the networks are designed by combining layer modules (CNN, RNN, etc), engineers less exploit the inductive bias, i.e., existing well-known rules or prior knowledge, other than annotated training data sets. We focus on employing sparsity-inducing priors in deep learning to encourage the network to concisely capture the nature of high-dimensional data in an unsupervised way. In order to use non-differentiable sparsity-inducing norms as loss functions, we plug their proximal mappings into the automatic differentiation framework. We demonstrate unsupervised learning of U-Net for background subtraction using low-rank and sparse priors. The U-Net can learn moving objects in a training sequence without any annotation, and successfully detect the foreground objects in test sequences.

show abstract

“…For instance, network quantization methods have been proposed for storage space compression by decreasing the number of possible and unique values for the parameters [13,22,23]. Tensor decomposition approaches decompose network matrices into several smaller ones to estimate the informative parameters of the deep CNNs with low-rank approximation/factorization [24,25,26,27]. More recently, [28] also propose a framework of architecture distillation based on layer-wise replacement, called LightweightNet for memory and time saving.…”

Section: Related Workmentioning

confidence: 99%

Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning

Yeom,

Seegerer,

Lapuschkin

et al. 2019

Preprint

View full text Add to dashboard Cite

The success of convolutional neural networks (CNNs) in various applications is accompanied by a significant increase in computation and parameter storage costs. Recent efforts to reduce these overheads involve pruning and compressing the weights of various layers while at the same time aiming to not sacrifice performance. In this paper, we propose a novel criterion for CNN pruning inspired by neural network interpretability: The most relevant elements, i.e. weights or filters, are automatically found using their relevance score in the sense of explainable AI (XAI). By that we for the first time link the two disconnected lines of interpretability and model compression research. We show in particular that our proposed method can efficiently prune transfer-learned CNN models where networks pre-trained on large corpora are adapted to specialized tasks. To this end, the method is evaluated on a broad range of computer vision datasets. Notably, our novel criterion is not only competitive or better compared to state-of-the-art pruning criteria when successive retraining is performed, but clearly outperforms these previous criteria in the common application setting where the data of the task to be transferred to are very scarce and no retraining is possible. Our method can iteratively compress the model while maintaining or even improving accuracy. At the same time, it has a computational cost in the order of gradient computation and is comparatively simple to apply without the need for tuning hyperparameters for pruning.

show abstract

Compressing by Learning in a Low-Rank and Sparse Decomposition Form

Abstract: Kailing Guo and Xiaona Xie contributed equally to this work.

Cited by 9 publications

References 21 publications

ResNet-like Architecture with Low Hardware Requirements

ResNet-like Architecture with Low Hardware Requirements

Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors

Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning

Contact Info

Product

Resources

About