Learning Low-Rank Structured Sparsity in Recurrent Neural Networks

Wu, Wen; Yang, Fan; Su, Yangfeng; Zhou, Dian; Zeng, Xuan

doi:10.1109/iscas45731.2020.9181239

Cited by 644 publications

(1,138 citation statements)

References 9 publications

Supporting

Mentioning

1,126

Contrasting

Unclassified

Order By: Relevance

“…For the former, the fluctuation of pruning ratio with the wide range change of hyperparameters in pruning LeNet models on MNIST dataset are investigated and then the parameters selection principle is summarised. For the latter, the proposed method is compared with some state‐of‐the‐art algorithms on CIFAR100 dataset, such as Liu's [24], ThiNet [15], SSL [23], SSR [31] and GM [20].…”

Section: Methodsmentioning

confidence: 99%

Providing clear pruning threshold: A novel CNN pruning method via L ₀ regularisation

Guo

2020

IET Image Processing

View full text Add to dashboard Cite

Network pruning is a significant way to improve the practicability of convolution neural networks (CNNs) by removing the redundant structure of the network model. However, in most of the existing network pruning methods l1 or l2 regularisation is applied to parameter matrices and the manual selection of pruning threshold is difficult and labor‐intensive. A novel CNNs network pruning method via l0 regularisation is proposed, which adopts l0 regularisation to expand the saliency gap between neurons. A half‐quadratic splitting (HQS) based iterative algorithm is put forward to calculate the approximation solution of l0 regularisation, which makes the joint optimisation problem of regularisation term and training loss function can be solved by various gradient‐based algorithms. Meanwhile, a hyperparameters selection method is designed to make most of the hyperparameters in the algorithm can be determined by examining the pre‐trained model. The results of experiments on MNIST, Fashion‐MNIST and CIFAR100 show that the proposed method can provide a much clearer pruning threshold by widening the saliency gap, and achieve a similar or even better compression performance, compared with the state‐of‐the‐art studies.

show abstract

Section: Methodsmentioning

confidence: 99%

Providing clear pruning threshold: A novel CNN pruning method via L ₀ regularisation

Guo

2020

IET Image Processing

View full text Add to dashboard Cite

show abstract

“…In the real‐valued case, a classical choice for the regulariser

r ()(\cdot)

in (6) is the

ℓ_{2}

norm. Whenever sparsity is desired, it can be replaced with the

ℓ_{1}

norm, or a proper group version is acting at a neuron‐level [16, 17]. In most implementations of QVNNs, these regularisers are applied element‐wise on the four components of each quaternion weight.…”

Section: Targeted Regularisation For Qvnnsmentioning

confidence: 99%

Compressing deep‐quaternion neural networks with targeted regularisation

Vecchi¹,

Scardapane²,

Comminiello³

et al. 2020

CAAI trans. intell. technol.

View full text Add to dashboard Cite

In recent years, hyper‐complex deep networks (such as complex‐valued and quaternion‐valued neural networks – QVNNs) have received a renewed interest in the literature. They find applications in multiple fields, ranging from image reconstruction to 3D audio processing. Similar to their real‐valued counterparts, quaternion neural networks require custom regularisation strategies to avoid overfitting. In addition, for many real‐world applications and embedded implementations, there is the need of designing sufficiently compact networks, with few weights and neurons. However, the problem of regularising and/or sparsifying QVNNs has not been properly addressed in the literature as of now. In this study, the authors show how to address both problems by designing targeted regularisation strategies, which can minimise the number of connections and neurons of the network during training. To this end, they investigate two extensions of ℓ1 and structured regularisations to the quaternion domain. In the authors’ experimental evaluation, they show that these tailored strategies significantly outperform classical (real‐valued) regularisation approaches, resulting in small networks especially suitable for low‐power and real‐time applications.

show abstract

“…In contrast, structured pruning enables to reduce the number of kernels or filters from CNNs [12]- [14] by using group sparse regularization such as group lasso [15], [16], whose method is widely used to pruning unnecessary kernels or filters keeping high performance.…”

Section: Arxiv:201102389v1 [Cscv] 4 Nov 2020mentioning

confidence: 99%

Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks

Mitsuno

Miyao

Kurita

2020

2020 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Since the convolutional neural networks are often trained with redundant parameters, it is possible to reduce redundant kernels or filters to obtain a compact network without dropping the classification accuracy. In this paper, we propose a filter pruning method using the hierarchical group sparse regularization. It is shown in our previous work that the hierarchical group sparse regularization is effective in obtaining sparse networks in which filters connected to unnecessary channels are automatically close to zero. After training the convolutional neural network with the hierarchical group sparse regularization, the unnecessary filters are selected based on the increase of the classification loss of the randomly selected training samples to obtain a compact network. It is shown that the proposed method can reduce more than 50% parameters of ResNet for CIFAR-10 with only 0.3% decrease in the accuracy of test samples. Also, 34% parameters of ResNet are reduced for TinyImageNet-200 with higher accuracy than the baseline network.

show abstract

Learning Low-Rank Structured Sparsity in Recurrent Neural Networks

Cited by 644 publications

References 9 publications

Providing clear pruning threshold: A novel CNN pruning method via L ₀ regularisation

Providing clear pruning threshold: A novel CNN pruning method via L ₀ regularisation

Compressing deep‐quaternion neural networks with targeted regularisation

Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks

Contact Info

Product

Resources

About

Learning Low-Rank Structured Sparsity in Recurrent Neural Networks

Cited by 644 publications

References 9 publications

Providing clear pruning threshold: A novel CNN pruning method via L 0 regularisation

Providing clear pruning threshold: A novel CNN pruning method via L 0 regularisation

Compressing deep‐quaternion neural networks with targeted regularisation

Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks

Contact Info

Product

Resources

About

Providing clear pruning threshold: A novel CNN pruning method via L ₀ regularisation

Providing clear pruning threshold: A novel CNN pruning method via L ₀ regularisation