Generalized Conditional Gradient with Augmented Lagrangian for Composite Minimization

Abstract

In this paper we propose a splitting scheme which hybridizes generalized conditional gradient with a proximal step which we call CGALP algorithm, for minimizing the sum of three proper convex and lower-semicontinuous functions in real Hilbert spaces. The minimization is subject to an affine constraint, that allows in particular to deal with composite problems (sum of more than three functions) in a separate way by the usual product space technique. While classical conditional gradient methods require Lipschitz-continuity of the gradient of the differentiable part of the objective, CGALP needs only differentiability (on an appropriate subset), hence circumventing the intricate question of Lipschitz continuity of gradients. For the two remaining functions in the objective, we do not require any additional regularity assumption. The second function, possibly nonsmooth, is assumed simple, i.e., the associated proximal mapping is easily computable. For the third function, again nonsmooth, we just assume that its domain is weakly compact and that a linearly perturbed minimization oracle is accessible. In particular, this last function can be chosen to be the indicator of a nonempty bounded closed convex set, in order to deal with additional constraints. Finally, the affine constraint is addressed by the augmented Lagrangian approach. Our analysis is carried out for a wide choice of algorithm parameters satisfying so called "open loop" rules. As main results, under mild conditions, we show asymptotic feasibility with respect to the affine constraint, boundedness of the dual multipliers, and convergence of the Lagrangian values to the saddle-point optimal value. We also provide (subsequential) rates of convergence for both the feasibility gap and the Lagrangian values.

Inexact and Stochastic Generalized Conditional Gradient with Augmented Lagrangian and Proximal Step

“…feasibility, Lagrangian convergence, and rates, are established. The analysis and results are far-reaching extensions of those in [34] to the inexact and stochastic setting, and require quite delicate new arguments. In Section 5 and Section 6, we consider different problem instances where inexact deterministic or stochastic computations are involved.…”

“…The primary contribution of this work is to analyze inexact and stochastic variants of the CGALP algorithm presented in [34] to address ( ). We coin this algorithm Inexact Conditional Gradient with Augmented Lagrangian and Proximal-step (ICGALP ).…”

“…

In this paper we propose and analyze inexact and stochastic versions of the CGALP algorithm developed in [34], which we denote ICGALP , that allow for errors in the computation of several important quantities. In particular this allows one to compute some gradients, proximal terms, and/or linear minimization oracles in an inexact fashion that facilitates the practical application of the algorithm to computationally intensive settings, e.g., in high (or possibly infinite) dimensional Hilbert spaces commonly found in machine learning problems.

…”

“…In fact, the problem under consideration here is exactly the same as that of [34], however, in this work, we consider an inexact extension of the algorithm presented and analyzed in [34] to solve ( ). The extension amounts to allowing either deterministic or stochastic errors in the computation of several quantities, including the gradient or proximal terms, e.g.…”

Scalable Semidefinite Programming

“…The standard CGM algorithm does not apply to the model problem (2.2) because of the affine constraint A X = b. Several variants [42,65,87,103] of CGM can handle affine constraints. In particular, CGAL [102] does so by applying CGM to an augmented Lagrangian formulation.…”