A derivative-free nonmonotone line-search technique for unconstrained optimization

Diniz-Ehrhardt, M. A.; Martínez, José Mário; Raydan, Marcos

doi:10.1016/j.cam.2007.07.017

Cited by 39 publications

(35 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…More related to our work is the contribution of [10], a derivative-free line search method that uses random searching directions, where at each iteration the step size is defined by a tolerant nonmonotone backtracking line search that always starts from the unitary step. The authors proved global convergence with probability one in [10], but no global rate was provided.…”

Section: Introductionmentioning

confidence: 99%

“…The authors proved global convergence with probability one in [10], but no global rate was provided. In addition, global convergence in [10] is much easier to establish than in our paper, since the backtracking line search always starts from the unitary step, independently of the previous iterations, and consequently the step size is little coupled to the history of the computation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Direct Search Based on Probabilistic Descent

Gratton¹,

Royer²,

Vicente³

et al. 2015

SIAM J. Optim.

126

View full text Add to dashboard Cite

Direct-search methods are a class of popular derivative-free algorithms characterized by evaluating the objective function using a step size and a number of (polling) directions. When applied to the minimization of smooth functions, the polling directions are typically taken from positive spanning sets which in turn must have at least n+1 vectors in an n-dimensional variable space. In addition, to ensure the global convergence of these algorithms, the positive spanning sets used throughout the iterations are required to be uniformly non-degenerate in the sense of having a positive (cosine) measure bounded away from zero.However, recent numerical results indicated that randomly generating the polling directions without imposing the positive spanning property can improve the performance of these methods, especially when the number of directions is chosen considerably less than n + 1.In this paper, we analyze direct-search algorithms when the polling directions are probabilistic descent, meaning that with a certain probability at least one of them is of descent type. Such a framework enjoys almost-sure global convergence. More interestingly, we will show a global decaying rate of 1/ √ k for the gradient size, with overwhelmingly high probability, matching the corresponding rate for the deterministic versions of the gradient method or of direct search. Our analysis helps to understand numerical behavior and the choice of the number of polling directions.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Direct Search Based on Probabilistic Descent

Gratton¹,

Royer²,

Vicente³

et al. 2015

SIAM J. Optim.

126

View full text Add to dashboard Cite

show abstract

“…( where X k :¼ X(ks) with k = 0, 1, 2,…, and step-size h would be selected appropriately via the line-search algorithm [8,[17][18][19] (presented in the ensuing section).…”

Section: Discrete-time Znn Modelmentioning

confidence: 99%

“…Different activation functions and different step sizes (which could be obtained via a line-search algorithm [8,[17][18][19]) are employed for such a model. When the linear activation function and step-size h:1 are used, the discrete-time ZNN model reduces exactly to Newton iteration for matrix square root finding.…”

Section: Introductionmentioning

confidence: 99%

Zhang neural network and its application to Newton iteration for matrix square root estimation

Zhang

Yang

Cai

et al. 2010

Neural Comput & Applic

View full text Add to dashboard Cite

A special class of recurrent neural networks (RNN) has recently been proposed by Zhang et al. for solving online time-varying matrix problems. Being different from conventional gradient-based neural networks (GNN), such RNN (termed specifically as Zhang neural networks, ZNN) are designed based on matrix-valued error functions, instead of scalar-valued norm-based energy functions. In this paper, we generalize and further investigate the ZNN model for time-varying matrix square root finding. For the purpose of possible hardware (e.g., digital circuit) realization, a discrete-time ZNN model is constructed and developed, which incorporates Newton iteration as a special case. Besides, to obtain an appropriate step-size value (in each iteration), a line-search algorithm is employed for the proposed discrete-time ZNN model. Computer-simulation results substantiate the effectiveness of the proposed ZNN model aided with a line-search algorithm, in addition to the connection and explanation to Newton iteration for matrix square root finding.

show abstract

“…As we already mentioned, some of the gradient free approaches, see for example [5,7], will probably make these algorithms more applicable in practice. Therefore, it will be the subject of future research.…”

Section: Numerical Results For Noisy Problemsmentioning

confidence: 99%

Line search methods with variable sample size

Krklec-Jerinkić¹

View full text Add to dashboard Cite

Minimization of unconstrained objective function in the form of mathematical expectation is considered. Sample Average Approximation -SAA method transforms the expectation objective function into a real-valued deterministic function using large sample and thus deals with deterministic function minimization. The main drawback of this approach is its cost. A large sample of the random variable that defines the expectation must be taken in order to get a reasonably good approximation and thus the sample average approximation method assumes very large number of function evaluations. We will present a line search strategy that uses variable sample size and thus makes the process significantly cheaper. Two measures of progress -lack of precision and decrease of function value are calculated at each iteration. Based on these two measures a new sample size is determined. The rule we will present allows us to increase or decrease the sample size in each iteration until we reach some neighborhood of the solution. An additional safeguard check is performed to avoid unproductive sample decrease. Eventually the maximal sample size is reached so the variable sample size strategy generates a solution of the same quality as * Department of Mathematics and Informatics, Faculty of Science, University of Novi Sad, Trg Dositeja Obradovića 4, 21000 Novi Sad, Serbia, e-mail: natasak@uns.ac.rs.

show abstract

A derivative-free nonmonotone line-search technique for unconstrained optimization

Cited by 39 publications

References 27 publications

Direct Search Based on Probabilistic Descent

Direct Search Based on Probabilistic Descent

Zhang neural network and its application to Newton iteration for matrix square root estimation

Line search methods with variable sample size

Contact Info

Product

Resources

About