Stochastic first-order methods for convex and nonconvex functional constrained optimization

Abstract

The monotone Variational Inequality (VI) is a general model that has important applications in various engineering and scientific domains. In numerous instances, the VI problems are accompanied by function constraints which can possibly be data-driven, making the usual projection operator challenging to compute. In this paper, we present novel first-order methods for the function constrained VI (FCVI) problem under various settings, including smooth or nonsmooth problems with stochastic operator and/or stochastic constraints. First, we introduce the OpConEx method and its stochastic variants, which employ extrapolation of the operator and constraint evaluations to update the variables and the Lagrangian multipliers. These methods achieve optimal operator or sample complexities when the FCVI problem is either (i) deterministic nonsmooth, or (ii) stochastic, including smooth or nonsmooth stochastic constraints. Notably, our algorithms are simple single-loop procedures and do not require the knowledge of Lagrange multipliers to attain these complexities. Second, to obtain the optimal operator complexity for smooth deterministic problems, we present a novel single-loop Adaptive Lagrangian Extrapolation (AdLagEx) method that can adaptively search for and explicitly bound the Lagrange multipliers. Moreover, we show that all of our algorithms can be easily extended to saddle point problems with coupled function constraints, hence achieving similar complexity results for the aforementioned cases. To our best knowledge, these complexities are obtained for the first time in the literature.

Iteration-complexity of an inexact proximal accelerated augmented Lagrangian method for solving linearly constrained smooth nonconvex composite optimization problems

“…Consider first the case in which z k is the exact solution of (5). Indeed, in this case, it follows from (6) and the optimality condition of (5) that 0 ∈ ∇f (z k ) + ∂h(z k ) + A * p k + (z k − z k−1 )/λ, and hence that (ẑ k , vk , pk ) := (z k , (z k−1 − z k )/λ, p k ) satisfies the above inclusion. Assume now that z k is an approximate solution of (5) in the sense that there exists a residual pair (v k , ε k ) which together with z k satisfies the approximation criterion (32) below.…”

“…Finally, the third one [6] studies a primal-dual proximal point type method for computing approximate stationary solution to a constrained smooth nonconvex composite optimization problem and establishes its iteration-complexity bounds under different sets of assumptions.…”

Conditional Gradient Methods for Convex Optimization with General Affine and Nonlinear Constraints

“…Our main contributions are briefly summarized as follows. Firstly, inspired by the constraint-extrapolation (ConEx) method for function constrained convex optimization in [4], we develop a novel constraint-extrapolated conditional gradient (CoexCG) method for solving problem (1.1). While both methods are single-loop primal-dual type methods for solving convex optimization problems with function constraints, CoexCG only requires us to minimize a linear function, rather than to perform projection, over X.…”

“…We also extend CoexDurCG for solving the aforementioned structured nonsmooth problems, and demonstrate that it is not necessary to explicitly define the smooth approximation problem. We note that this technique of adding a diminishing regularization term can be applied for solving problems with either unbounded primal feasible region (e.g., stochastic subgradient descent [29] and stochastic accelerated gradient descent [19]), or unbounded dual feasible region (e.g., ConEx [4]), for which one often requires the number of iterations fixed in advance.…”

“…Our development has been inspired the constraint extrapolation (ConEx) method recently introduced by Boob, Deng and Lan [4] for solving problem (2.6). ConEx is an accelerated primal-dual type method which updates both the primal variable x and dual variables (y, z) in the each iteration.…”