Fast CP‐compression layer: Tensor CP‐decomposition to compress layers in deep learning

Abstract

Deep neural network (DNN) shows its powerful performance in terms of image classification and many other applications. However, as the number of network layers increases, it brings huge pressure on devices with limited resources. In this article, a novel network compression algorithm is proposed that compresses the original network by up to about 60 times. In particular, a tensor Canonical Polyadic(CP) decomposition based algorithm is proposed to compress the weight matrix in the fully connected(FC) layer and the convolution kernel in the convolution layer. Traditional tensor decomposition algorithms are usually to first pre‐train the weights, and decompose the weights, finally perform fine‐tuning on the factors in the second training phase. Instead, the decomposed factors are directly updated by performing tensor CP decomposition on weight without fine‐tuning. The proposed algorithm is called Fast CP‐Compression Layer method in this paper. Experiments show that the proposed algorithm cannot only reduce computing time and improve compression factor but also improve accuracy on some datasets.

HMC: Hybrid model compression method based on layer sensitivity grouping

“…To verify the superiority of the proposed method, HMC is compared with the single compression algorithm and the additive hybrid compression algorithm, where tensor decomposition algorithms include Tucker [ 43 ], CP [ 44 ], TT [ 45 ], and MUSCO [ 28 ]. Structured pruning algorithms include Hrank [ 16 ], CHEX [ 17 ], DepGraph [ 18 ]; The additive hybrid compression algorithm includes literature [ 41 ] and ATMC [ 38 ].…”

“…To verify the superiority of the proposed method, HMC is compared with the single compression algorithm and the additive hybrid compression algorithm, where tensor decomposition algorithms include Tucker [43], CP [44], TT [45], and MUSCO [28]. Structured pruning…”