The non-convex geometry of low-rank matrix optimization

Abstract

This work considers two popular minimization problems: (i) the minimization of a general convex function f (X) with the domain being positive semi-definite matrices; (ii) the minimization of a general convex function f (X) regularized by the matrix nuclear norm X * with the domain being general matrices. Despite their optimal statistical performance in the literature, these two optimization problems have a high computational complexity even when solved using tailored fast convex solvers. To develop faster and more scalable algorithms, we follow the proposal of Burer and Monteiro to factor the low-rank variable X = UU (for semi-definite matrices) or X = UV (for general matrices) and also replace the nuclear norm X * with ( U 2 F + V 2 F )/2. In spite of the non-convexity of the resulting factored formulations, we prove that each critical point either corresponds to the global optimum of the original convex problems or is a strict saddle where the Hessian matrix has a strictly negative eigenvalue. Such a nice geometric structure of the factored formulations allows many local search algorithms to find a global optimizer even with random initializations.

Toward the Optimal Construction of a Loss Function Without Spurious Local Minima for Solving Quadratic Equations

“…The reader can refer to the review paper [8] for more details. Geometric landscape of related loss functions for low rank matrix recovery has been investigated in [18,19,28,4,32]. Similar results have also been established for nonconvex formulations of other problems, for example blind deconvolution [49], dictionary learning [34,35], tensor completion [1,20], phase synchronization [5,29,6], and deep neural networks [39,47,33,25].…”

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

“…In particular, [160] derived a model-free theory where no assumptions are imposed on M , which accommodates, for example, the noisy case and the case where the truth is only approximately low-rank. The global landscape of asymmetric matrix sensing / completion holds similarly by considering a loss function regularized by the term g(L, R) := L L − R R 2 F [159,173] or by the term g(L, R) := L 2 F + R 2 F [31]. Theorem 34 has been generalized to the asymmetric case in [40].…”

Nonconvex Robust Low-Rank Matrix Recovery

“…Key to these results is certain error bound conditions, which elucidate the regularity properties of the underlying optimization problem. Recently, the above results have been extended to cover general smooth low-rank matrix optimization problems whose objective functions satisfy the restricted strong convexity and smoothness properties [29,52,53].…”