ACSL: Adaptive correlation-driven sparsity learning for deep neural network compression

He, Wei; Wu, Meiqing; Lam, Siew-Kei

doi:10.1016/j.neunet.2021.09.012

Cited by 6 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The neural networks, we know, can approximate any function, which can accomplish a predetermined goal. Consequently, various types neural networks have been applied in a large number of fields (see earlier studies [1][2][3][4][5][6][7][8]). The application of integrated circuit chip with neuron unit to the field of artificial intelligence is a good example.…”

Section: Introductionmentioning

confidence: 99%

Stability analysis based on a control adjuster for switched neural networks by trajectory similarity

Zhou

Zhang

2023

Math Methods in App Sciences

View full text Add to dashboard Cite

This paper focuses on a class of exponential stability control problem for the stochastic neural networks (SNNs) with time‐varying delays and Markov jump (TVDMJ). As a prerequisite to the main theorem, the existence and uniqueness of the solution for the main system are proved via contraction map principle. For the stability control of the system, we design a controller which can be adjusted the parameters to make the system stabilization. We use this controller to actively control the stability behavior of the system, rather than the papers by setting the appropriate parameters to make the system achieve a stable state. Based on the intermittent control, we design a novel trajectory similarity process control and obtain exponential stability results, which are less conservative than the existing theoretical results. Two examples based on some numerical calculation and simulation diagrams are presented to validate the effectiveness for the utilized techniques.

show abstract

Section: Introductionmentioning

confidence: 99%

Stability analysis based on a control adjuster for switched neural networks by trajectory similarity

Zhou

Zhang

2023

Math Methods in App Sciences

View full text Add to dashboard Cite

show abstract

“…Chapter 5 is submitted to journal in [2] as Wei He, Meiqing Wu, Siew-Kei Lam. The explicit contributions of the co-authors are listed below:…”

Section: Authorship Attribution Statementmentioning

confidence: 99%

“…The work described in this chapter has been submitted for publication as [2] Algorithm 1 ACSL for channel pruning.…”

Section: Chapter 5 Adaptive Correlation-driven Sparsity Learning For ...mentioning

confidence: 99%

Deep neural network compression for pixel-level vision tasks

He¹

Self Cite

View full text Add to dashboard Cite

Deep convolutional neural networks (DCNNs) have demonstrated remarkable performance in many computer vision tasks. In order to achieve this, DCNNs typically require a large number of trainable parameters that are optimized to extract informative features. This often results in over-parameterization of the DCNN models, which incurs high computational complexity and large storage requirements that hinder their deployment on embedded devices with stringent computational and memory resources. In this thesis, we aim to develop DCNN compression methods to generate compact DCNN models that can still produce comparable performance as the original models. In particular, our proposed methods must lend themselves well towards DCNN models for pixel-level vision tasks (such as semantic segmentation and crowd counting). DCNN compression for pixel-level vision tasks has not been thoroughly investigated, as existing works mainly target the less challenging image-level classification task.We first present a framework that utilizes knowledge distillation to recover the performance loss of DCNN models that have undergone network pruning. This departs from the existing knowledge distillation approaches, where the student model and teacher model are pre-defined before knowledge adaptation. Experiments on the encoder-decoder type models for semantic segmentation demonstrate that the proposed framework can effectively recover the performance loss of the compact student model after aggressive pruning in most cases. However, in certain cases, knowledge transfer cannot outperform the conventional fine-tuning process on the pruned semantic segmentation architectures.Next, we propose Context-Aware Pruning (CAP) that utilizes channel association, which captures the contextual information, to exploit parameters redundancy for pruning semantic segmentation models. We evaluated our framework on widely-used benchmarks and showed its effectiveness on both large and lightweight models. Our framework reduces v the number of parameters of state-of-the-art semantic segmentation models PSPNet-101, PSPNet-50, ICNet, and SegNet, by 32%, 47%, 54%, and 63% respectively on the Cityscapes dataset. This reduction is achieved while preserving the best performance among all the baselines pruning methods considered. Finally, we propose Adaptive Correlation-driven Sparsity Learning (ACSL) for DCNN compression that can provide superior performance on both image-level and pixel-level vision tasks. ACSL extends CAP by inducing sparsity into the channel importance with an adaptive penalty strength. The experimental results demonstrate that ACSL outperforms state-of-the-art pruning methods on image-level classification, semantic segmentation, and dense crowd counting tasks. In particular for the crowd counting task, the proposed ACSL framework is able to reduce the DCNN model parameters by up to 94%, while maintaining the same performance of (at times outperforming) the original model. vi Firstly, I would like to express my deepest gratitude to my supervisor Prof. Si...

show abstract

A novel sequential structure for lightweight multi-scale feature learning under limited available images

Liu

Vong

2023

Neural Networks

View full text Add to dashboard Cite

ACSL: Adaptive correlation-driven sparsity learning for deep neural network compression

Cited by 6 publications

References 34 publications

Stability analysis based on a control adjuster for switched neural networks by trajectory similarity

Stability analysis based on a control adjuster for switched neural networks by trajectory similarity

Deep neural network compression for pixel-level vision tasks

A novel sequential structure for lightweight multi-scale feature learning under limited available images

Contact Info

Product

Resources

About