An Optimal Statistical and Computational Framework for Generalized Tensor Estimation

Abstract

This paper describes a flexible framework for generalized low-rank tensor estimation problems that includes many important instances arising from applications in computational imaging, genomics, and network analysis. The proposed estimator consists of finding a lowrank tensor fit to the data under generalized parametric models. To overcome the difficulty of non-convexity in these problems, we introduce a unified approach of projected gradient descent that adapts to the underlying low-rank structure. Under mild conditions on the loss function, we establish both an upper bound on statistical error and the linear rate of computational convergence through a general deterministic analysis. Then we further consider a suite of generalized tensor estimation problems, including sub-Gaussian tensor denoising, tensor regression, and Poisson and binomial tensor PCA. We prove that the proposed algorithm achieves the minimax optimal rate of convergence in estimation error. Finally, we demonstrate the superiority of the proposed framework via extensive experiments on both simulated and real data.

Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurements

“…For instance, while nuclear norm minimization achieves near-optimal statistical guarantees for low-rank matrix estimation [CT10] with a polynomial run time, computing the nuclear norm of a tensor turns out to be NP-hard [FL18]. Therefore, there have been a number of efforts to develop polynomialtime algorithms for tensor recovery, including but not limited to the sum-of-squares hierarchy [BM16,PS17], nuclear norm minimization with unfolding [GRY11,MHWG14], regularized gradient descent [HWZ20], to name a few; see Section 1.4 for further discussions.…”

“…Despite a flurry of activities for understanding factored gradient descent in the matrix setting [CLC19], this line of algorithmic thinkings has been severely under-explored for the tensor setting, especially when it comes to provable guarantees for both sample and computational complexities. The closest existing theory that one comes across is [HWZ20] for tensor regression, which adds regularization terms to promote the orthogonality of the factors U , V , W :…”

“…Here, α > 0 and β > 0 are two parameters to be specified. While encouraging, theoretical guarantees of this regularized GD algorithm [HWZ20] still fall short of achieving computational efficiency. In truth, its convergence speed is rather slow: it takes an order of κ 2 log(1/ε) iterations to attain an ε-accurate estimate of the ground truth tensor, where κ is a sort of condition number of X ⋆ to be defined momentarily.…”

“…n 2 r polynomial tensor [MHWG14] Projected GD n 2 r κ 2 log 1 ε tensor [CRY19] Regularized GD n 3/2 rκ 4 κ 2 log 1 ε factor [HWZ20] Riemannian Gauss-Newton n 3/2 r 3/2 κ 4 log log 1 ε tensor [LZ21] (concurrent) 5 ScaledGD n 3/2 rκ 2 log 1 ε factor (this paper) Table 2: Comparisons of ScaledGD with existing algorithms for tensor regression when the tensor is low-rank under the Tucker decomposition. Here, we say that the output X of an algorithm reaches ε-accuracy, if it satisfies X − X ⋆ F ≤ εσ min (X ⋆ ).…”

“…Figure 1 illustrates the number of iterations needed to achieve a relative error X − X ⋆ F ≤ 10 −3 X ⋆ F for ScaledGD and regularized GD [HWZ20] under different condition numbers for tensor completion under the Bernoulli sampling model (see Section 4 for experimental settings). Clearly, the iteration complexity of regularized GD [HWZ20] deteriorates at a super linear rate with respect to the condition number κ, while ScaledGD enjoys an iteration complexity that is independent of κ as predicted by our theory. Indeed, with a seemingly small modification, ScaledGD takes merely 17 iterations to achieve the desired accuracy over the entire range of κ, while regularized GD takes thousands of iterations even with a moderate condition number!…”