Local and Global Linear Convergence of General Low-rank Matrix Recovery Problems

Abstract

We study the convergence rate of gradient-based local search methods for solving low-rank matrix recovery problems with general objectives in both symmetric and asymmetric cases, under the assumption of the restricted isometry property. First, we develop a new technique to verify the Polyak-Łojasiewicz inequality in a neighborhood of the global minimizers, which leads to a local linear convergence region for the gradient descent method. Second, based on the local convergence result and a sharp strict saddle property proven in this paper, we present two new conditions that guarantee the global linear convergence of the perturbed gradient descent method. The developed local and global convergence results provide much stronger theoretical guarantees than the existing results.Preprint. Under review.

Sharp Analysis of Stochastic Optimization under Global Kurdyka-Łojasiewicz Inequality

“…It is worth noting that in this setting, O( −1 f ) is optimal [30]. 7 Stationary point of a dynamic is its convergence point.…”

“…Understanding the dynamics of this recursion, allows us to establish the global convergence of SGD. Our approach consists of two main steps: i) Finding the stationary 7 point of (6) when the inequality is replaced by equality and for a fixed step-size, η k = η, which we denote by r(η). ii) Selecting the step-sizes {η k } and sequence {b k } k≥0 , such that the corresponding stationary points {r(η k )} k≥0 (defined below) converge to zero as k increases.…”

“…This is function is both positive and φ(t) 2 is convex. Next, we apply the result of Theorem 1 to obtain the convergence rate of SGD for this type of loss functions assuming the stochastic gradient estimator satisfying Assumption 4 with h(t) = t. First step is to obtain the stationary point r(η) using Equation (7).…”

“…To do so, we first characterize the stationary point for the special setting of this corollary. Equation (7) becomes…”

“…The special case of KŁ condition, 2-PŁ, often referred as Polyak-Łojasiewicz or PŁ condition, was originally discovered independently in the seminal works of B. Polyak, T. Ležanski and S. Łojasiewicz [48,33,38,39]. Notably, this class of problems has found many interesting emerging applications in machine learning, for instance, policy gradient (PG) methods in reinforcement learning [41,1,56], generalized linear models [40], over-parameterized neural networks [2,57], linear quadratic regulator in optimal control [13,19], and low-rank matrix recovery [7].…”