An adaptive stochastic sequential quadratic programming with differentiable exact augmented lagrangians

Na, Sen; Anitescu, Mihai; Kolar, Mladen

doi:10.1007/s10107-022-01846-z

Cited by 21 publications

(52 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, a class of stochastic SQP methods has been developed for solving (1.1). These methods outperform stochastic penalty methods empirically and have convergence guarantees in expectation [7,28]. In [7], the authors propose an objective-function-free stochastic SQP method with adaptive step sizes for the fully stochastic regime.…”

Section: Introductionmentioning

confidence: 99%

“…In [7], the authors propose an objective-function-free stochastic SQP method with adaptive step sizes for the fully stochastic regime. In contrast, in [28], the authors propose a stochastic step search (referred to as line search in the paper [28]) SQP method for the setting in which the errors in the function and derivative approximations can be diminished. We note that several algorithm choices in the two papers [7,28], e.g., merit functions and merit parameters, are different.…”

Section: Introductionmentioning

confidence: 99%

“…In contrast, in [28], the authors propose a stochastic step search (referred to as line search in the paper [28]) SQP method for the setting in which the errors in the function and derivative approximations can be diminished. We note that several algorithm choices in the two papers [7,28], e.g., merit functions and merit parameters, are different. Several other extensions have been proposed [3,6,8,17,27,32], and very few of these works (or others in the literature) derive worst-case iteration complexity (or sample complexity) due to the difficulties that arise because of the constrained setting and the stochasticity.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Sequential Quadratic Optimization for Nonlinear Equality Constrained Stochastic Optimization

Berahas¹,

Curtis²,

Robinson³

et al. 2021

SIAM J. Optim.

View full text Add to dashboard Cite

A step-search sequential quadratic programming method is proposed for solving nonlinear equality constrained stochastic optimization problems. It is assumed that constraint function values and derivatives are available, but only stochastic approximations of the objective function and its associated derivatives can be computed via inexact probabilistic zeroth-and first-order oracles. Under reasonable assumptions, a high-probability bound on the iteration complexity of the algorithm to approximate first-order stationarity is derived. Numerical results on standard nonlinear optimization test problems illustrate the advantages and limitations of our proposed method.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sequential Quadratic Optimization for Nonlinear Equality Constrained Stochastic Optimization

Berahas¹,

Curtis²,

Robinson³

et al. 2021

SIAM J. Optim.

View full text Add to dashboard Cite

show abstract

“…One of the future works is to apply our Hessian averaging technique on constrained nonlinear optimization problems. [45,46] designed various stochastic second-order methods based on sequential quadratic programming (SQP) for solving constrained problems, where the Hessian of the Lagrangian function was estimated by subsampling. These works established the global convergence for stochastic SQP methods, while the local convergence rate of these methods remains unknown.…”

Section: Discussionmentioning

confidence: 99%

Hessian averaging in stochastic Newton methods achieves superlinear convergence

Dereziński

Mahoney

2022

Math. Program.

Self Cite

View full text Add to dashboard Cite

We consider minimizing a smooth and strongly convex objective function using a stochastic Newton method. At each iteration, the algorithm is given an oracle access to a stochastic estimate of the Hessian matrix. The oracle model includes popular algorithms such as Subsampled Newton and Newton Sketch, which can efficiently construct stochastic Hessian estimates for many tasks, e.g., training machine learning models. Despite using second-order information, these existing methods do not exhibit superlinear convergence, unless the stochastic noise is gradually reduced to zero during the iteration, which would lead to a computational blow-up in the per-iteration cost. We propose to address this limitation with Hessian averaging: instead of using the most recent Hessian estimate, our algorithm maintains an average of all the past estimates. This reduces the stochastic noise while avoiding the computational blow-up. We show that this scheme exhibits local Q-superlinear convergence with a non-asymptotic rate of $$(\varUpsilon \sqrt{\log (t)/t}\,)^{t}$$ ( Υ log ( t ) / t ) t , where $$\varUpsilon $$ Υ is proportional to the level of stochastic noise in the Hessian oracle. A potential drawback of this (uniform averaging) approach is that the averaged estimates contain Hessian information from the global phase of the method, i.e., before the iterates converge to a local neighborhood. This leads to a distortion that may substantially delay the superlinear convergence until long after the local neighborhood is reached. To address this drawback, we study a number of weighted averaging schemes that assign larger weights to recent Hessians, so that the superlinear convergence arises sooner, albeit with a slightly slower rate. Remarkably, we show that there exists a universal weighted averaging scheme that transitions to local convergence at an optimal stage, and still exhibits a superlinear convergence rate nearly (up to a logarithmic factor) matching that of uniform Hessian averaging.

show abstract

“…Non-convex minimization problems with non-convex constraints have been recently studied due to their popularity in the machine learning community [16,14]. [57,58] developed stochastic algorithms based on sequential quadratic programming that incorporate a stochastic line search and converge globally to a first-order stationary point of the optimization program. [18] and [1] used a Lagrangian-based approach with access to an optimization oracle and found a distribution over solutions rather than a pure equilibrium.…”

Section: Related Workmentioning

confidence: 99%

Provably training overparameterized neural network classifiers with non-convex constraints

Chen

Wang

Kolar

2022

Electron. J. Statist.

Self Cite

View full text Add to dashboard Cite

Training a classifier under non-convex constraints has gotten increasing attention in the machine learning community thanks to its wide range of applications such as algorithmic fairness and class-imbalanced classification. However, several recent works addressing non-convex constraints have only focused on simple models such as logistic regression or support vector machines. Neural networks, one of the most popular models for classification nowadays, are precluded and lack theoretical guarantees. In this work, we show that overparameterized neural networks could achieve a near-optimal and near-feasible solution of non-convex constrained optimization problems via the project stochastic gradient descent. Our key ingredient is the no-regret analysis of online learning for neural networks in the overparameterization regime, which may be of independent interest in online learning applications.

show abstract

An adaptive stochastic sequential quadratic programming with differentiable exact augmented lagrangians

Cited by 21 publications

References 41 publications

Sequential Quadratic Optimization for Nonlinear Equality Constrained Stochastic Optimization

Sequential Quadratic Optimization for Nonlinear Equality Constrained Stochastic Optimization

Hessian averaging in stochastic Newton methods achieves superlinear convergence

Provably training overparameterized neural network classifiers with non-convex constraints

Contact Info

Product

Resources

About