Generalized Higher Order Orthogonal Iteration for Tensor Learning and Decomposition

Liu, Yuanyuan; Shang, Fanhua; Fan, Wei; Cheng, James; Cheng, Hong

doi:10.1109/tnnls.2015.2496858

Cited by 88 publications

(72 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another line of research focuses on the convex relaxation of the low rank tensor decomposition and completion problem. For example, the convex relaxation is achieved by a generalized trace norm (Romera‐Paredes and Pontil, ), a tensor Schatten 1‐norm (Liu et al ., ; Gu et al ., ) or a novel tensor nuclear norm (Yuan and Zhang, ). Kolda and Bader (), Plantenga et al .…”

Section: Introductionmentioning

confidence: 97%

Provable Sparse Tensor Decomposition

Sun

Han

et al. 2016

Journal of the Royal Statistical Society Series B: Statistical Methodology

109

View full text Add to dashboard Cite

Summary We propose a novel sparse tensor decomposition method, namely the tensor truncated power method, that incorporates variable selection in the estimation of decomposition components. The sparsity is achieved via an efficient truncation step embedded in the tensor power iteration. Our method applies to a broad family of high dimensional latent variable models, including high dimensional Gaussian mixtures and mixtures of sparse regressions. A thorough theoretical investigation is further conducted. In particular, we show that the final decomposition estimator is guaranteed to achieve a local statistical rate, and we further strengthen it to the global statistical rate by introducing a proper initialization procedure. In high dimensional regimes, the statistical rate obtained significantly improves those shown in the existing non‐sparse decomposition methods. The empirical advantages of tensor truncated power are confirmed in extensive simulation results and two real applications of click‐through rate prediction and high dimensional gene clustering.

show abstract

Section: Introductionmentioning

confidence: 97%

Provable Sparse Tensor Decomposition

Sun

Han

et al. 2016

Journal of the Royal Statistical Society Series B: Statistical Methodology

109

View full text Add to dashboard Cite

show abstract

“…Since then, HOSVD and HOOI have been widely studied in the literature (see, e.g. [30,31,32,33,34]). However as far as we know, many basic theoretical properties of these procedures, such as the error bound and the necessary iteration times, still remain unclear.…”

Section: Introductionmentioning

confidence: 99%

Tensor SVD: Statistical and Computational Limits

Zhang

Dong

2018

IEEE Trans. Inform. Theory

158

133

View full text Add to dashboard Cite

In this paper, we propose a general framework for tensor singular value decomposition (tensor SVD), which focuses on the methodology and theory for extracting the hidden lowrank structure from high-dimensional tensor data. Comprehensive results are developed on both the statistical and computational limits for tensor SVD. This problem exhibits three different phases according to the signal-to-noise ratio (SNR). In particular, with strong SNR, we show that the classical higher-order orthogonal iteration achieves the minimax optimal rate of convergence in estimation; with weak SNR, the information-theoretical lower bound implies that it is impossible to have consistent estimation in general; with moderate SNR, we show that the non-convex maximum likelihood estimation provides optimal solution, but with NP-hard computational cost; moreover, under the hardness hypothesis of hypergraphic planted clique detection, there are no polynomial-time algorithms performing consistently in general.[28] may yield sub-optimal result. We will further illustrate this phenomenon by numerical analysis in Section 5.Remark 2. Especially when r = 1, Theorem 1 confirms the heuristic conjecture raised in Richard and Montanari [23] that the tensor unfolding method yields reliable estimates for order-3 spiked tensors provided that λ/σ > O(p 3/4 ). Moreover, Theorem 1 further shows the power iterations are necessary in order to refine the reliable estimates to minimax-optimal estimates. Our result in Theorem 1 outperforms the ones by Sum-of-Squares (SOS) scheme (see, e.g., [24,36]), where an additional logarithm factor on the assumption of λ is required. In addition, the method we analyze here, i.e., HOOI, is efficient, easy to implement, and achieves optimal rate of convergence for estimation error.Remark 3. The strong SNR assumption (λ/σ ≥ Cp 3/4 ) is crucial to guarantee the performance of Algorithm 1. Actually, to ensure that Step 1 in Algorithm 1 provides meaningful initializations, λ should be at least of order p 3/4 according to our theoretical analysis.Moreover, the estimators with high likelihood, such as MLE, achieve the following upper bounds under weaker assumption that λ/σ ≥ Cp 1/2 .

show abstract

“…Schatten norm, or trace norm), which is defined as the sum of singular values of a matrix and it is the most popular convex surrogate for rank regularization. Based on differ-ent definitions of tensor rank, various nuclear norm regularized algorithms have been proposed (Liu et al 2013;Imaizumi, Maehara, and Hayashi 2017;Liu et al 2014;Liu et al 2015). Rank minimization based methods do not need to specify the rank of the employed tensor decompositions beforehand, and the rank of the recovered tensor will be automatically learned from the limited observations.…”

Section: Introductionmentioning

confidence: 99%

Tensor Ring Decomposition with Rank Minimization on Latent Space: An Efficient Approach for Tensor Completion

Yuan

Mandic

et al. 2019

AAAI

136

View full text Add to dashboard Cite

In tensor completion tasks, the traditional low-rank tensor decomposition models suffer from the laborious model selection problem due to their high model sensitivity. In particular, for tensor ring (TR) decomposition, the number of model possibilities grows exponentially with the tensor order, which makes it rather challenging to find the optimal TR decomposition. In this paper, by exploiting the low-rank structure of the TR latent space, we propose a novel tensor completion method which is robust to model selection. In contrast to imposing the low-rank constraint on the data space, we introduce nuclear norm regularization on the latent TR factors, resulting in the optimization step using singular value decomposition (SVD) being performed at a much smaller scale. By leveraging the alternating direction method of multipliers (ADMM) scheme, the latent TR factors with optimal rank and the recovered tensor can be obtained simultaneously. Our proposed algorithm is shown to effectively alleviate the burden of TR-rank selection, thereby greatly reducing the computational cost. The extensive experimental results on both synthetic and real-world data demonstrate the superior performance and efficiency of the proposed approach against the state-of-the-art algorithms.

show abstract

Generalized Higher Order Orthogonal Iteration for Tensor Learning and Decomposition

Cited by 88 publications

References 43 publications

Provable Sparse Tensor Decomposition

Provable Sparse Tensor Decomposition

Tensor SVD: Statistical and Computational Limits

Tensor Ring Decomposition with Rank Minimization on Latent Space: An Efficient Approach for Tensor Completion

Contact Info

Product

Resources

About