Heuristic rank selection with progressively searching tensor ring network

Li, Nannan; Pan, Yu; Chen, Yaran; Ding, Zixiang; Zhao, Dan; Xu, Zenglin

doi:10.1007/s40747-021-00308-x

Cited by 24 publications

(32 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tensorized decomposition methods substitute a high-dimensional tensor with the product of multiple low-dimensional tensors. These are Tucker decomposition (TD), canonical polyadic decomposition (CPD), tensor train (TT), and tensor ring (TR) in the tensor decomposition [ 25 , 26 , 27 , 28 ]. In tensor decomposition, the lower the rank of decomposed tensors, the lower the amount of computation, but the lower the accuracy of the network.…”

Section: Related Researchmentioning

confidence: 99%

Simplification of Deep Neural Network-Based Object Detector for Real-Time Edge Computing

Choi

Jung

et al. 2023

Sensors

View full text Add to dashboard Cite

This paper presents a method for simplifying and quantizing a deep neural network (DNN)-based object detector to embed it into a real-time edge device. For network simplification, this paper compares five methods for applying channel pruning to a residual block because special care must be taken regarding the number of channels when summing two feature maps. Based on the comparison in terms of detection performance, parameter number, computational complexity, and processing time, this paper discovers the most satisfying method on the edge device. For network quantization, this paper compares post-training quantization (PTQ) and quantization-aware training (QAT) using two datasets with different detection difficulties. This comparison shows that both approaches are recommended in the case of the easy-to-detect dataset, but QAT is preferable in the case of the difficult-to-detect dataset. Through experiments, this paper shows that the proposed method can effectively embed the DNN-based object detector into an edge device equipped with Qualcomm’s QCS605 System-on-Chip (SoC), while achieving a real-time operation with more than 10 frames per second.

show abstract

Section: Related Researchmentioning

confidence: 99%

Simplification of Deep Neural Network-Based Object Detector for Real-Time Edge Computing

Choi

Jung

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…The previous study [18] uses two TTD methods in the experiments, but the results are very close without any indepth analysis. Even tensor-ring, a Train-TTD-like SOTA method, reports that it suffers from a 0.88% accuracy drop with a 2.2× compression ratio for ResNet32 on Cifar-10 [29]. Moreover, accuracy degradation can be worse on deep MLPs because their structures and blocks essentially differ from RNNs and CNNs.…”

Section: B Tensor-train For Deep Neural Networkmentioning

confidence: 99%

TT-MLP: Tensor Train Decomposition on Deep MLPs

Yan

Ando

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Deep multilayer perceptrons (MLPs) have achieved promising performance on computer vision tasks. Deep MLPs consist solely of fully-connected layers as the conventional MLPs do but adopt more sophisticated network architectures based on mixer layers composed of token-mixing and channelmixing components. These architectures enable deep MLPs to have global receptive fields, but the significant increase of parameters becomes a massive burden on practical applications. To tackle this problem, we focus on using tensor-train decomposition (TTD) for compressing deep MLPs. At first, this paper analyzes deep MLPs under conventional TTD methods, especially using various designs of a macro framework and micro blocks: The former is how to concatenate mixer layers, and the latter is how to design a mixer layer. Based on the analysis, we propose a novel TTD method named Train-TTD-Train. The proposed method exerts the learning capability of channel-mixing components and improves the trade-off between accuracy and size. In the evaluation, the proposed method showed a better trade-off than conventional TTD methods on ImageNet-1K and achieved a 0.56% higher inference accuracy with a 15.44% memory reduction on Cifar-10.INDEX TERMS Tensor-train decomposition, low-rank approximation, deep neural networks, deep multilayer perceptron, network parameter compression.

show abstract

“…Classic DNNs can be classified into five categories based on different compression techniques [1] : pruning [2] , quantization [3] , knowledge distillation [4] , low-rank decomposition [5] , and network architecture search [6] . Low-rank decomposition allows for extracting Low-rank features from original tensor by tensor decomposition.…”

Section: Introductionmentioning

confidence: 99%

DNN compression approach based on Bayesian optimization tensor ring decomposition

Zhang,

Liu,

Shi

et al. 2024

Fifteenth International Conference on Graphics and Image Processing (ICGIP 2023)

View full text Add to dashboard Cite

Tensor ring (TR) decomposition is an effective method to achieve deep neural network (DNN) compression. However, there are two problems with TR decomposition: setting TR rank to equal in TR decomposition and selecting rank through an iterative process is time-consuming. To address the two problems, A TR network compression method by Bayesian optimization (TR-BO) is proposed. TR-BO involves selecting rank via Bayesian optimization, compressing the neural network layer via TR decomposition using rank obtained in the previous step, and, finally, further fine-tuning the compressed model to overcome some of the performance loss due to compression. Experimental results show that TR-BO achieves the best results in terms of Top-1 accuracy, parameter, and training time. For example, on the CIFAR-10 dataset Resnet20 network, TR-BO-1 achieves 87.67% accuracy with a compression ratio of 13.66 and a running time of only 2.4 hours. Furthermore, TR-BO has achieved state-of-the-art performance on the CIFAR-10/100 benchmark tests.

show abstract

Heuristic rank selection with progressively searching tensor ring network

Cited by 24 publications

References 19 publications

Simplification of Deep Neural Network-Based Object Detector for Real-Time Edge Computing

Simplification of Deep Neural Network-Based Object Detector for Real-Time Edge Computing

TT-MLP: Tensor Train Decomposition on Deep MLPs

DNN compression approach based on Bayesian optimization tensor ring decomposition

Contact Info

Product

Resources

About