Finding stationary points on bounded-rank matrices: a geometric hurdle and a smooth remedy

Abstract

We consider the problem of provably finding a stationary point of a smooth function to be minimized on the variety of bounded-rank matrices. This turns out to be unexpectedly delicate. We trace the difficulty back to a geometric obstacle: On a nonsmooth set, there may be sequences of points along which standard measures of stationarity tend to zero, but whose limit points are not stationary. We name such events apocalypses, as they can cause optimization algorithms to converge to non-stationary points. We illustrate this explicitly for an existing optimization algorithm on bounded-rank matrices. To provably find stationary points, we modify a trust-region method on a standard smooth parameterization of the variety. The method relies on the known fact that second-order stationary points on the parameter space map to stationary points on the variety. Our geometric observations and proposed algorithm generalize beyond boundedrank matrices. We give a geometric characterization of apocalypses on general constraint sets, which implies that Clarke-regular sets do not admit apocalypses. Such sets include smooth manifolds, manifolds with boundaries, and convex sets. Our trust-region method supports parameterization by any complete Riemannian manifold.

The effect of smooth parametrizations on nonconvex optimization landscapes

“…It is therefore important to characterize "robust" versions of "k ⇒ 1", guaranteeing that approximate k-critical points for (Q) map to approximate stationary points for (P). Note that if X lacks regularity, care is needed when defining approximate stationarity for (P), see [33].…”

“…For an example of scenario (a), consider minimizing a cost f over the set X = R m×n ≤r of all m × n matrices of rank at most r. Unfortunately, standard algorithms running on (P) may converge to a non-stationary point because of the nonsmooth geometry of X [33,37]. Instead of trying to solve (P) directly, it is common to parametrize X by the linear space M = R m×r ×R n×r using the rank factorization ϕ(L, R) = LR ⊤ , and to solve (Q) instead.…”

Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss

“…We will transfer the results obtained on the space of linear maps M k to the space of covariance matrices S + (k, n). Borrowing the terminology from (Levin et al, 2022), we introduce the following notations and definitions. Let M be any smooth manifold, E a linear space, ϕ : M → E a smooth (over)parametrization (or lift) of the search space X = ϕ(M) ⊆ E. The following problems are considered…”

First-order optimization on stratified sets

“…If x ∈ C is a local minimizer of f | C , then x is stationary for (1), hence Mordukhovich stationary for (1), and hence Clarke stationary for (1). The stationarity of a point depends only on f | C since, by [35,Lemmas A.7 and A.8], the correspondence…”

“…A frequently encountered obstacle against guaranteeing convergence to stationary points of (1) is the possible presence in C of so-called apocalyptic points. By [35,Definition 2.7], a point x ∈ C is said to be apocalyptic if there exist a sequence (x i ) i∈N in C converging to x and a continuously differentiable function φ : E → R such that lim i→∞ s(x i ; φ, C) = 0 whereas s(x; φ, C) > 0. Such a triplet (x, (x i ) i∈N , φ) is called an apocalypse.…”