A Stochastic Line Search Method with Expected Complexity Analysis

Paquette, Courtney; Scheinberg, Katya

doi:10.1137/18m1216250

Cited by 88 publications

(93 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, there are other applications where Assumption 3.2 is not satisfied, as is the case when errors are due to Gaussian noise. Nevertheless, since the analysis for unbounded errors appears to be complex [6], we will not consider it here, as our main goal is to advance our understanding of the BFGS method in the presence of errors, and this is best done, at first, in a benign setting.…”

Section: Algorithm 21 Outline Of the Bfgs Methods With Errorsmentioning

confidence: 99%

Analysis of the BFGS Method with Errors

Xie¹,

Byrd

Nocedal

2020

SIAM J. Optim.

View full text Add to dashboard Cite

The classical convergence analysis of quasi-Newton methods assumes that the function and gradients employed at each iteration are exact. In this paper, we consider the case when there are (bounded) errors in both computations and establish conditions under which a slight modification of the BFGS algorithm with an Armijo-Wolfe line search converges to a neighborhood of the solution that is determined by the size of the errors. One of our results is an extension of the analysis presented in [4], which establishes that, for strongly convex functions, a fraction of the BFGS iterates are good iterates. We present numerical results illustrating the performance of the new BFGS method in the presence of noise.[16]. We view these as less desirable alternatives for reasons discussed in the next section. The line search strategy could also be performed in other ways. For example, in their analysis of a gradient method, Berahas et al. [2], relax the Armijo conditions to take noise into account. We prefer to retain the standard Armijo-Wolfe line search without any modification, as this has practical advantages.The literature of the BFGS method with inaccurate gradients includes the implicit filtering method of Kelley et al. [5,10], which assumes that noise can be diminished at will at any iteration. Deterministic convergence guarantees have been established for that method by ensuring that noise decays as the iterates approach the solution. Dennis and Walker [7] and Ypma [18] study bounded deterioration properties, and local convergence, of quasi-Newton methods with errors, when started near the solution with a Hessian approximation that is close to the exact Hessian. Barton [1] proposes an implementation of the BFGS method in which gradients are computed by an appropriate finite differencing technique, assuming that the noise level in the function evaluation is known. Berahas et al. [2] estimate the noise in the function using Hamming's finite difference technique [9], as extended by Moré and Wild [11], and employ this estimate to compute a finite difference gradient in the BFGS method. They analyze a gradient method with a relaxation of the Armijo condition, and do not study the effects of noise in BFGS updating.There has recently been some interest in designing quasi-Newton methods for machine learning applications using stochastic approximations to the gradient [3,8,12,17]. These papers avoid potential difficulties with BFGS or L-BFGS updating by assuming that the quality of gradient differences is always controlled, and as a result, the analysis follows similar lines as for classical BFGS and L-BFGS.This paper is organized in 5 sections. The proposed algorithm is described in Section 2. Section 3, the bulk of the paper, presents a sequence of lemmas related to the existence of stepsizes that satisfy the Armijo-Wolfe conditions, the beneficial effect of lengthening the differencing interval, the properties of "good iterates", culminating in a global convergence result. Some numerical tests that illustrate the performance ...

show abstract

Section: Algorithm 21 Outline Of the Bfgs Methods With Errorsmentioning

confidence: 99%

Analysis of the BFGS Method with Errors

Xie¹,

Byrd

Nocedal

2020

SIAM J. Optim.

View full text Add to dashboard Cite

show abstract

“…This framework, in principal, can be used for analysis of convergence rates of a variety of algorithms -for instance it applies to all algorithms in [6] and [16]. In recent work [24] it has been applied to analyze a stochastic line-search method. In this paper, specifically, we use this general framework to derive a bound on the convergence rate of the STORM algorithm defined in [8], by proving that these assumptions are satisfied by this algorithm.…”

Section: Introductionmentioning

confidence: 99%

Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales

Blanchet

Cartis

Menickelly

et al. 2019

INFORMS Journal on Optimization

Self Cite

103

161

View full text Add to dashboard Cite

We propose a novel framework for analyzing convergence rates of stochastic optimization algorithms with adaptive step sizes. This framework is based on analyzing properties of an underlying generic stochastic process, in particular by deriving a bound on the expected stopping time of this process. We utilize this framework to analyze the bounds on expected global convergence rates of a stochastic variant of a traditional trust region method, introduced in [8]. While traditional trust region methods rely on exact computations of the gradient, Hessian and values of the objective function, this method assumes that these values are available up to some dynamically adjusted accuracy. Moreover, this accuracy is assumed to hold only with some sufficiently large, but fixed, probability, without any additional restrictions on the variance of the errors. This setting applies, for example, to standard stochastic optimization and machine learning formulations. Improving upon the analysis in [8], we show that the stochastic process defined by the algorithm satisfies the assumptions of our proposed general framework, with the stopping time defined as reaching accuracy ∇f (x) ≤ ǫ. The resulting bound for this stopping time is O(ǫ −2 ), under the assumption of sufficiently accurate stochastic gradient, and is the first global complexity bound for a stochastic trust-region method. Finally, we apply the same framework to derive second order complexity bound under some additional assumptions.

show abstract

“…It does not provide an algorithm which is robust against failures to satisfy the adaptive accuracy requirements. This is in contrast with the interesting analysis of unconstrained firstorder methods of [25] and [8]. Combining the generality of our approach with the robustness of the proposal in these latter papers is thus desirable.…”

Section: Discussionmentioning

confidence: 99%

Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization

et al. 2019

View full text Add to dashboard Cite

A regularization algorithm using inexact function values and inexact derivatives is proposed and its evaluation complexity analyzed. This algorithm is applicable to unconstrained problems and to problems with inexpensive constraints (that is constraints whose evaluation and enforcement has negligible cost) under the assumption that the derivative of highest degree is β-Hölder continuous. It features a very flexible adaptive mechanism for determining the inexactness which is allowed, at each iteration, when computing objective function values and derivatives. The complexity analysis covers arbitrary optimality order and arbitrary degree of available approximate derivatives. It extends results of Cartis, Gould and Toint [Sharp worst-case evaluation complexity bounds for arbitraryorder nonconvex optimization with inexpensive constraints, arXiv:1811.01220, 2018] on the evaluation complexity to the inexact case: if a q-th order minimizer is sought using approximations to the first p derivatives, it is proved that a suitable approximate minimizer within ǫ is computed by the proposed algorithm in at most O ǫ − p+β p−q+β iterations and at most O | log(ǫ)|ǫ − p+β p−q+β approximate evaluations. An algorithmic variant, although more rigid in practice, can be proved to find such an approximate minimizer in O | log(ǫ)| + ǫ − p+β p−q+β evaluations. While the proposed framework remains so far conceptual for high degrees and orders, it is shown to yield simple and computationally realistic inexact methods when specialized to the unconstrained and bound-constrained first-and second-order cases. The deterministic complexity results are finally extended to the stochastic context, yielding adaptive sample-size rules for subsampling methods typical of machine learning. Bellavia, Gurioli, Morini, Toint: Adaptive Regularization Algorithms with Inexact Evaluations2riving formal bounds on the number of evaluations of the objective function (and possibly of its derivatives) necessary to obtain approximate optimal solutions within a user-specified accuracy. Until recently, the results had focused on methods using first-and second-order derivatives of the objective function, and on convergence guarantees to first-or second-order stationary points [29,23,24,19,11]. Among these contributions, [24,11] analyzed the "regularization method", in which a model of the objective function around a given iterate is constructed by adding a regularization term to the local Taylor expansion, model which is then approximately minimized in an attempt to find a new point with a significantly lower objective function value [21]. Such methods have been shown to possess optimal evaluation complexity [14] for first-and second-order models and minimizers, and have generated considerable interest in the research community. A theoretically significant step was made in [7] for unconstrained problems, where evaluation complexity bounds were obtained for convergence to first-order stationary points of a simplified regularization method using models of arbitrary degree...

show abstract

A Stochastic Line Search Method with Expected Complexity Analysis

Cited by 88 publications

References 15 publications

Analysis of the BFGS Method with Errors

Analysis of the BFGS Method with Errors

Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales

Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization

Contact Info

Product

Resources

About