The effect of smooth parametrizations on nonconvex optimization landscapes

Abstract

We develop new tools to study the landscapes of nonconvex optimization problems. Specifically, we leverage the fact that many such problems can be paired with others via a smooth parametrization of their domains. The global optima of such pairs of problems correspond to each other, but their landscapes can be significantly different. We introduce a framework to relate the two landscapes. Applications include: optimization over low-rank matrices and tensors by optimizing over a factorization; the Burer-Monteiro approach to semidefinite programs; training neural networks by optimizing over their weights and biases; and quotienting out symmetries. In all these examples, one of the two problems is smooth, and hence can be tackled with algorithms for optimization on manifolds. These can find desirable points (e.g., critical points). We determine the properties that ensure these map to desirable points for the other problem via the smooth parametrization. These new tools enable us to strengthen guarantees for an array of optimization problems, previously obtained on a case-bycase basis in the literature.

First-order optimization on stratified sets

“…In Section 3.1, we list several papers using the concept of stratified set in optimization. Then, in Section 3.2, we review [35, Algorithm 1] and, more generally, the main results of [34] which concern optimization through a smooth lift. Finally, in Section 3.3, we review the RFDR algorithm.…”

“…In [34], the authors study problem (1) under the assumption that there exist a smooth manifold M and a smooth map ϕ : M → E such that ϕ(M) = C; they call ϕ a smooth lift of C. Specifically, they investigate how desirable points of the problem min y∈M (f • ϕ)(y) (5) map to desirable points of (1):…”

“…• [34,Theorem 2.8] gives a necessary and sufficient condition on ϕ for the property "for all continuous f , if y ∈ M is a local minimum of (5), then ϕ(y) is a local minimum of (1)" to hold;…”

“…• [34,Theorem 2.10] implies that, for every y ∈ M, if T C (ϕ(y)) is not a linear subspace of E, then there exists a differentiable f such that y is a stationary point of (5) and ϕ(y) is not a stationary point of (1);…”

“…The iteration map of the proposed algorithm applies a step of projected-projected gradient descent with backtracking line search, as proposed by Schneider and Uschmajew (2015), to its input but also to a projection of the input onto each of the lower strata to which it is considered close, and outputs a point among those thereby produced that maximally reduces the cost function. Under our assumptions on the stratified set, we prove that this algorithm produces a sequence whose accumulation points are stationary, and therefore does not follow the so-called apocalypses described by Levin, Kileel, and Boumal (2022). We illustrate the apocalypse-free property of our method through a numerical experiment on the determinantal variety.…”