A Novel Rank Selection Scheme in Tensor Ring Decomposition Based on Reinforcement Learning for Deep Neural Networks

Cheng, Zhiyu; Li, Baopu; Fan, Yanwen; Bao, Yingze

doi:10.1109/icassp40776.2020.9053292

Cited by 25 publications

(22 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yerlan et al [28] formulate the low-rank compression problem as a mixed discrete-continuous optimization jointly over the rank elements and over the matrix elements. Zhiyu et al [2] propose a novel rank selection scheme for tensor ring, which apply deep deterministic policy gradient to control the selection of rank. Their algorithms calculate the optimal rank directly from the trained weight matrix without the analysis of rank.…”

Section: Rank Selectionmentioning

confidence: 99%

“…Experimental setting The dimension of TR-RseNet is shown in Table 6, Ψ is the number of ResBlock. TR-ResNet32 is built as introduced by Wenqi et al [26] with Ψ as 5, and TR-ResNet20 is constructed as proposed by Zhiyu et al [2] with Ψ as 3. First, we apply the PSTRN-M/S to search TR-ResNet20/32 on CIFAR10.…”

Section: Experiments On Cifar10 and Cifar100mentioning

confidence: 99%

“…Tucker [7] and TT [5] are works that compress neural networks by other tensor decomposition methods. TR-RL [2] search the rank of TR-based model based on reinforcement learning. The first block compares PSTRN-M with the results of low rank decomposition works that have fewer parameters.…”

Section: Experiments On Cifar10 and Cifar100mentioning

confidence: 99%

See 2 more Smart Citations

Heuristic rank selection with progressively searching tensor ring network

Pan

Chen

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

Recently, tensor ring networks (TRNs) have been applied in deep networks, achieving remarkable successes in compression ratio and accuracy. Although highly related to the performance of TRNs, rank selection is seldom studied in previous works and usually set to equal in experiments. Meanwhile, there is not any heuristic method to choose the rank, and an enumerating way to find appropriate rank is extremely time-consuming. Interestingly, we discover that part of the rank elements is sensitive and usually aggregate in a narrow region, namely an interest region. Therefore, based on the above phenomenon, we propose a novel progressive genetic algorithm named progressively searching tensor ring network search (PSTRN), which has the ability to find optimal rank precisely and efficiently. Through the evolutionary phase and progressive phase, PSTRN can converge to the interest region quickly and harvest good performance. Experimental results show that PSTRN can significantly reduce the complexity of seeking rank, compared with the enumerating method. Furthermore, our method is validated on public benchmarks like MNIST, CIFAR10/100, UCF11 and HMDB51, achieving the state-of-the-art performance.

show abstract

Section: Rank Selectionmentioning

confidence: 99%

Section: Experiments On Cifar10 and Cifar100mentioning

confidence: 99%

See 1 more Smart Citation

Heuristic rank selection with progressively searching tensor ring network

Pan

Chen

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…Hence, the selection of TD models can also be viewed as a topology searching process, which is a discrete optimization problem analogous to the layer-wise architecture search in convolutional neural networks [50]. From (5), we can observe that the edge rank determines the topology and complexity of Ψ. Supposing that all the elements in R Ψ are equal to R and…”

Section: (F)mentioning

confidence: 99%

“…Since the rank determination for a fixed topology is NP-hard [49]. Cheng et al [50] adopt reinforcement learning (RL) to determine the rank of TR representation and applied it to neural network compression tasks. In [51], the TR rank is gradually increased based on the measured sensitivity of the approximation error of factor.…”

mentioning

confidence: 99%

Multi-Tensor Network Representation for High-Order Tensor Completion

Nie,

Wang,

Lai

2021

Preprint

View full text Add to dashboard Cite

This work studies the problem of high-dimensional data (referred to as tensors) completion from partially observed samplings. We consider that a tensor is a superposition of multiple low-rank components. In particular, each component can be represented as multilinear connections over several latent factors and naturally mapped to a specific tensor network (TN) topology. In this paper, we propose a fundamental tensor decomposition (TD) framework: Multi-Tensor Network Representation (MTNR), which can be regarded as a linear combination of a range of TD models, e.g., CANDECOMP/PARAFAC (CP) decomposition, Tensor Train (TT), and Tensor Ring (TR). Specifically, MTNR represents a high-order tensor as the addition of multiple TN models, and the topology of each TN is automatically generated instead of manually pre-designed. For the optimization phase, an adaptive topology learning (ATL) algorithm is presented to obtain latent factors of each TN based on a rank incremental strategy and a projection error measurement strategy. In addition, we theoretically establish the fundamental multilinear operations for the tensors with TN representation, and reveal the structural transformation of MTNR to a single TN. Finally, MTNR is applied to a typical task, tensor completion, and two effective algorithms are proposed for the exact recovery of incomplete data based on the Alternating Least Squares (ALS) scheme and Alternating Direction Method of Multiplier (ADMM) framework. Extensive numerical experiments on synthetic data and real-world datasets demonstrate the effectiveness of MTNR compared with the start-of-the-art methods.

show abstract