Tensor Regression Using Low-Rank and Sparse Tucker Decompositions

Ahmed, Talal; Raja, Haroon; Bajwa, Waheed U.

doi:10.1137/19m1299335

Cited by 17 publications

(10 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[CRY19] proposed projected gradient descent algorithms with respect to the tensors, which have larger computation and memory footprints than the factored gradient descent approaches taken in this paper. [ARB20] proposed a tensor regression model where the tensor is simultaneously low-rank and sparse in the Tucker decomposition. A concurrent work [LZ21] proposed a Riemannian Gauss-Newton algorithm, and obtained an impressive quadratic convergence rate for tensor regression (see Table 2).…”

Section: Additional Related Workmentioning

confidence: 99%

Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurements

Tong¹,

Ma²,

Prater-Bennette³

et al. 2021

Preprint

View full text Add to dashboard Cite

Tensors, which provide a powerful and flexible model for representing multi-attribute data and multiway interactions, play an indispensable role in modern data science across various fields in science and engineering. A fundamental task is to faithfully recover the tensor from highly incomplete measurements in a statistically and computationally efficient manner. Harnessing the low-rank structure of tensors in the Tucker decomposition, this paper develops a scaled gradient descent (ScaledGD) algorithm to directly recover the tensor factors with tailored spectral initializations, and shows that it provably converges at a linear rate independent of the condition number of the ground truth tensor for two canonical problemstensor completion and tensor regression -as soon as the sample size is above the order of n 3/2 ignoring other parameter dependencies, where n is the dimension of the tensor. This leads to an extremely scalable approach to low-rank tensor estimation compared with prior art, which suffers from at least one of the following drawbacks: extreme sensitivity to ill-conditioning, high per-iteration costs in terms of memory and computation, or poor sample complexity guarantees. To the best of our knowledge, ScaledGD is the first algorithm that achieves near-optimal statistical and computational complexities simultaneously for low-rank tensor completion with the Tucker decomposition. Our algorithm highlights the power of appropriate preconditioning in accelerating nonconvex statistical estimation, where the iteration-varying preconditioners promote desirable invariance properties of the trajectory with respect to the underlying symmetry in low-rank tensor factorization.

show abstract

Section: Additional Related Workmentioning

confidence: 99%

Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurements

Tong¹,

Ma²,

Prater-Bennette³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Here, we use a mode sparsity constraint, which induces the sparsity of each element of the CP-decomposition for every activation tensors independently. In tensor regression, this regularization is also used in [36] for instance.…”

Section: Tensor Convolutional Dictionary Learning With Cp Low-rank Ac...mentioning

confidence: 99%

Tensor Convolutional Dictionary Learning With CP Low-Rank Activations

Humbert

Oudre

Vayatis

et al. 2022

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

In this paper, we propose an extension of the standard CDL problem with tensor representation, where each activation is constrained to be "low-rank" through a Canonical Polyadic decomposition. We show that this important additional constraint increases the robustness of the CDL with respect to strong noise and improve the interpretability of the results. Additionally, we discuss in details the benefits of this representation. Then, we propose two new algorithms, based on respectively ADMM or FISTA, that efficiently solve this problem, by leveraging the low-rank property and achieve a lower complexity than the leading CDL algorithms. Finally, we evaluate our approach on a wide range of experiments, highlighting the modularity and the important advantages of this tensorial lowrank formulation.

show abstract

“…First, the low-rank tensor estimation has attracted much recent attention from machine learning and statistics communities. Various methods were proposed, including the convex relaxation (Mu et al, 2014;Raskutti et al, 2019;Tomioka et al, 2011), projected gradient descent (Rauhut et al, 2017;Chen et al, 2019a;Ahmed et al, 2020;Yu and Liu, 2016), gradient descent on the factorized model (Han et al, 2020b;Cai et al, 2019;Hao et al, 2020), alternating minimization (Zhou et al, 2013;Jain and Oh, 2014;Liu and Moitra, 2020;Xia et al, 2020), and importance sketching (Zhang et al, 2020a). Moreover, when the target tensor has order two, our problem reduces to the widely studied low-rank matrix recovery/estimation (Recht et al, 2010;Li et al, 2019;Ma et al, 2019;Sun and Luo, 2015;Tu et al, 2016;Wang et al, 2017;Zhao et al, 2015;Zheng and Lafferty, 2015;Charisopoulos et al, 2021;Luo et al, 2020;Bauch et al, 2021).…”

Section: Related Literaturementioning

confidence: 99%

“…In both tensor regression and SVD, the estimation upper bounds in Theorems 3 and 4 match the lower bounds in the literature ((Zhang and Xia, 2018, Theorem 3) and (Zhang et al, 2020a, Theorem 5)), which shows that RGN achieves the minimax optimal rate of estimation error. Compared to existing algorithms in the literature on tensor regression and SVD (Ahmed et al, 2020;Chen et al, 2019a;Han et al, 2020b;Zhang and Xia, 2018), RGN is the first to achieve the minimax rate-optimal estimation error with only a double-logarithmic number of iterations attributed to its second-order convergence. Suppose d, r are fixed and the condition number κ is of order Op1q, we note that the sample size requirement (n ě Op ?…”

Section: Tensor Svdmentioning

confidence: 99%

“…Alternatively, a large body of literature turn to the non-convex formulation and focus on developing computational efficient twostage procedures on estimating X ˚: first, one obtains a warm initialization of X ˚and then runs local algorithms to refine the estimate. Provable guarantees on estimation or recovery of X ˚for such a two-stage paradigm have been developed in different scenarios (Rauhut et al, 2017;Chen et al, 2019a;Ahmed et al, 2020;Han et al, 2020b;Cai et al, 2019;Hao et al, 2020;Cai et al, 2020;Xia et al, 2020). In this work, we focus on the non-convex formulation and aim to develop a provable computationally efficient estimator for X ˚.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Low-rank Tensor Estimation via Riemannian Gauss-Newton: Statistical Optimality and Second-Order Convergence

Luo

Zhang²

2021

Preprint

View full text Add to dashboard Cite

In this paper, we consider the estimation of a low Tucker rank tensor from a number of noisy linear measurements. The general problem covers many specific examples arising from applications, including tensor regression, tensor completion, and tensor PCA/SVD. We propose a Riemannian Gauss-Newton (RGN) method with fast implementations for low Tucker rank tensor estimation. Different from the generic (super)linear convergence guarantee of RGN in the literature, we prove the first quadratic convergence guarantee of RGN for low-rank tensor estimation under some mild conditions. A deterministic estimation error lower bound, which matches the upper bound, is provided that demonstrates the statistical optimality of RGN. The merit of RGN is illustrated through two machine learning applications: tensor regression and tensor SVD. Finally, we provide the simulation results to corroborate our theoretical findings.

show abstract

Tensor Regression Using Low-Rank and Sparse Tucker Decompositions

Cited by 17 publications

References 30 publications

Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurements

Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurements

Tensor Convolutional Dictionary Learning With CP Low-Rank Activations

Low-rank Tensor Estimation via Riemannian Gauss-Newton: Statistical Optimality and Second-Order Convergence

Contact Info

Product

Resources

About