A novel compact design of convolutional layers with spatial transformation towards lower-rank representation for image classification

Liu, Baichen; Han, Zhi; Chen, Xi’ai; Shao, Wang; Jia, Huidi; Wang, Yanmei; Tang, Yandong

doi:10.1016/j.knosys.2022.109723

Cited by 4 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To validate the effectiveness of our approach, we compare it with several recent low-rank tensor decomposition approaches, vector quantization approaches, scalar quantization approaches and network pruning methods. The low-rank tensor decomposition approaches include SVD [42], LCT [34], TDNR [43], HALOC [44], Maestro [45] and ELRT [46]. The vector quantization approaches include PQF [16] and BGD [47].…”

Section: Methodsmentioning

confidence: 99%

“…The authors of [33] showed that, with a suitable formulation, determining the optimal rank of each layer was amenable to a mixed discrete continuous optimization jointly over the ranks and matrix elements. The authors of [34] proposed a novel compact design of convolutional layers with a spatial transformation towards a lower-rank representation, and they applied trainable spatial transformations to low-rank convolutional kernels in a predefined Tucker product form to enhance the versatility of convolutional kernels. These approaches can improve the identification accuracy of a compressed network to some extent.…”

Section: Related Workmentioning

confidence: 99%

“…Table 3 shows the experimental results of ResNet18 on the ImageNet dataset with our approach and the approaches used for comparison. The HALOC [44], LCT [34], and ELRT [46] approached the aim to compress the parameter count while maintaining identification accuracy and, thus, only achieved a compression ratio of less than 3×. Direct low-rank decomposition and pruning approaches such as TRP [32] and SVD [42] were not able to achieve high compression ratios, and their accuracy noticeably decreased.…”

Section: Imagenet Experimentsmentioning

confidence: 99%

See 2 more Smart Citations

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Liu,

Wang,

et al. 2024

Electronics

Self Cite

View full text Add to dashboard Cite

Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network’s performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Imagenet Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Liu,

Wang,

et al. 2024

Electronics

Self Cite

View full text Add to dashboard Cite

show abstract

“…To validate the advancement and effectiveness of the TR-BO method, TR-BO result are compared with seven stateof-the-art compression methods (i.e., TRN [12] , TR-RL [8] , PSTRN [5] , TRP [13] , LC [14] , LCT [15] , LCCUR [16] ) and show good performance on two datasets with two networks, as shown in Tables 2 and 3 (Bold represents best performance). Compared to the PSTRN and TR-RL methods, the proposed TR-BO method achieves optimal performance in terms of accuracy, number of parameters, and training time at any compression ratio.…”

Section: Image Classificationmentioning

confidence: 99%

DNN compression approach based on Bayesian optimization tensor ring decomposition

Zhang,

Liu,

Shi

et al. 2024

Fifteenth International Conference on Graphics and Image Processing (ICGIP 2023)

View full text Add to dashboard Cite

Tensor ring (TR) decomposition is an effective method to achieve deep neural network (DNN) compression. However, there are two problems with TR decomposition: setting TR rank to equal in TR decomposition and selecting rank through an iterative process is time-consuming. To address the two problems, A TR network compression method by Bayesian optimization (TR-BO) is proposed. TR-BO involves selecting rank via Bayesian optimization, compressing the neural network layer via TR decomposition using rank obtained in the previous step, and, finally, further fine-tuning the compressed model to overcome some of the performance loss due to compression. Experimental results show that TR-BO achieves the best results in terms of Top-1 accuracy, parameter, and training time. For example, on the CIFAR-10 dataset Resnet20 network, TR-BO-1 achieves 87.67% accuracy with a compression ratio of 13.66 and a running time of only 2.4 hours. Furthermore, TR-BO has achieved state-of-the-art performance on the CIFAR-10/100 benchmark tests.

show abstract

Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian Approach

Liu,

Zhang,

Shi

et al. 2024

Neural Process Lett

View full text Add to dashboard Cite

Due to deep neural networks (DNNs) a large number of parameters, DNNs increase the demand for computing and storage during training, reasoning and deployment, especially when DNNs stack deeper and wider. Tensor decomposition can not only compress DNN models but also reduce parameters and storage requirements while maintaining high accuracy and performance. About tensor ring (TR) decomposition of tensor decomposition, there are two problems: (1) The practice of setting the TR rank to be equal in TR decomposition results in an unreasonable rank configuration. (2) The training time of selecting rank through iterative processes is time-consuming. To address the two problems, a TR network compression method by Variational Bayesian (TR-VB) is proposed based on the Global Analytic Solution of Empirical Variational Bayesian Matrix Factorization (GAS of EVBMF). The method consists of three steps: (1) rank selection, (2) TR decomposition, and (3) fine-tuning to recover accumulated loss of accuracy. Experimental results show that, for a given network, TR-VB gives the best results in terms of Top-1 accuracy, parameters, and training time under different compression levels. Furthermore, TR-VB validated on CIFAR-10/100 public benchmarks achieves state-of-the-art performance.

show abstract

A novel compact design of convolutional layers with spatial transformation towards lower-rank representation for image classification

Cited by 4 publications

References 21 publications

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

DNN compression approach based on Bayesian optimization tensor ring decomposition

Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian Approach

Contact Info

Product

Resources

About