Nonconvex Lagrangian-Based Optimization: Monitoring Schemes and Global Convergence

Abstract

We introduce a novel approach addressing global analysis of a difficult class of nonconvexnonsmooth optimization problems within the important framework of Lagrangian-based methods. This genuine nonlinear class captures many problems in modern disparate fields of applications. It features complex geometries, qualification conditions, and other regularity properties do not hold everywhere. To address these issues we work along several research lines to develop an original general Lagrangian methodology which can deal, all at once, with the above obstacles. A first innovative feature of our approach is to introduce the concept of Lagrangian sequences for a broad class of algorithms. Central to this methodology is the idea of turning an arbitrary descent method into a multiplier method. Secondly, we provide these methods with a transitional regime allowing us to identify in finitely many steps a zone where we can tune the step-sizes of the algorithm for the final converging regime. Then, despite the min-max nature of Lagrangian methods, using an original Lyapunov method we prove that each bounded sequence generated by the resulting monitoring schemes are globally convergent to a critical point for some fundamental Lagrangian-based methods in the broad semialgebraic setting, which to the best of our knowledge, are the first of this kind.

ADMM for multiaffine constrained optimization

“…A method derived from ADMM has also been proposed for optimizing a biaffine model for training deep neural networks [53]. For general nonlinear constraints, a framework for "monitored" Lagrangian-based multiplier methods was studied in [4].…”

A Proximal Minimization Algorithm for Structured Nonconvex and Nonsmooth Problems

“…Splitting algorithms for solving problems of the form (1.2) have been considered in [19], under the assumption that H is twice continuously differentiable with bounded Hessian, in [25], under the assumption that one of the summands is convex and continuous on its effective domain, and in [13], as a particular case of a general nonconvex proximal ADMM algorithm. We would like to mention in this context also [10] for the case when A is nonlinear. The convergence analysis we will carry out in this paper relies on a descent inequality, which we prove for a regularization of the augmented Lagrangian L β : R mˆRqˆRpˆRp Ñ R Y t`8u L β px, y, z, uq " F pzq`G pyq`H px, yq`xu, Ax´zy`β 2 Ax´z 2 , β ą 0, associated with problem (1.1).…”

Composite Optimization by Nonconvex Majorization-Minimization

“…Note that the minimizer of this majorizer can not only be computed efficiently due to its separability, but also allows for a global view of the function and its minimizer almost coincides with the global minimum although the initial point is quite far from it. Finally, a recent preprint [15] proposes to solve composite minimization problems with a different approach, namely a nonlinear splitting variant, reformulating the problem to…”

“…Critically both ours and their approach rely on the efficient solution of a nonlinear programming task as intermediate step in the algorithm. For us, this is the nonconvex majorizer (3), the corresponding problem in [15,Eq. (6.3)] is the minimization of (8) for u:…”

“…In practice this algorithm is applicable even if the subproblems (15) can only be solved up to a local optimum. However it is especially interesting if (15) can actually be solved globally.…”