Nonconvex Low-Rank Tensor Completion from Noisy Data

Abstract

This paper investigates a problem of broad practical interest, namely, the reconstruction of a large-dimensional low-rank tensor from highly incomplete and randomly corrupted observations of its entries. Although a number of papers have been dedicated to this tensor completion problem, prior algorithms either are computationally too expensive for large-scale applications or come with suboptimal statistical performance. Motivated by this, we propose a fast two-stage nonconvex algorithm—a gradient method following a rough initialization—that achieves the best of both worlds: optimal statistical accuracy and computational efficiency. Specifically, the proposed algorithm provably completes the tensor and retrieves all low-rank factors within nearly linear time, while at the same time enjoying near-optimal statistical guarantees (i.e., minimal sample complexity and optimal estimation accuracy). The insights conveyed through our analysis of nonconvex optimization might have implications for a broader family of tensor reconstruction problems beyond tensor completion.

Landscape Correspondence of Empirical and Population Risks in the Eigendecomposition Problem

“…Proposition 1. [24, Propositions 4.6.1, 4.6.2] For a symmetric matrix M ∈ R N ×N , a vector x ∈ R N with x 2 = 1 is an eigenvector of M if and only if it is a critical point of the population risk (5) 2 . Moreover, denote {λ n } N n=1 with λ 1 > λ 2 ≥ λ 3 ≥ • • • ≥ λ N as the eigenvalues of M and {v n } N n=1 as the associated eigenvectors.…”

“…For any small constant δ ∈ (0, 1], it follows from [17, Lemma 13] that Y−EY ≤ δ x 2 2 with probability at least 1 − c 1 N −c2 provided that m ≥ Cδ −2 N log(N ) for some constant C. Therefore, by setting = Cδ x 2 2 ≤ η = 0.11d min , we can conclude that the two conditions in (7) and (8) hold with probability at least 1 − c 1 N −c2 as long as the number of measurements satisfies m ≥ Cd −2 min x 4 2 N log(N ). Moreover, it follows from Corollary III.1 that the distance between the empirical local minimum and population local minimum is on the order of d −1 min δ x 2 2 .…”

“…10 Note that we assume x, v 1 ≥ 0 here. In the case when x, v 1 < 0, one can bound x + v 1 2 2 instead.…”

“…S PECTRAL methods are of fundamental importance in signal processing and machine learning due to their simplicity and effectiveness. They have been widely used to extract useful information from noisy and partially observed data in a variety of applications, including dimensionality reduction [1], tensor estimation [2], ranking from pairwise comparisons [3], low-rank matrix estimation [4], and community detection [5], to name a few. As is well known, the theoretical performance of the gradient descent algorithm and its variants is heavily dependent on a proper initialization.…”

Initialization and Regularization of Factorized Neural Layers

“…For the former, several papers use SI after training (Nakkiran et al, 2015;Yaguchi et al, 2019;Yang et al, 2020), while Ioannou et al (2016) argue for initializing factors as though they were single layers, which we find inferior to SI in some cases. Outside deep learning, spectral methods have also been shown to yield better initializations for certain matrix and tensor problems (Keshavan et al, 2010;Cai et al, 2019). For regularization, Gray et al (2019) suggest compression-rate scaling (CRS), which scales weight-decay using the reduction in parameter count; this is justified via the usual Bayesian understanding of 2 -regularization (Murphy, 2012).…”