A class of gradient unconstrained minimization algorithms with adaptive stepsize

Abstract

In this paper the development, convergence theory and numerical testing of a class of gradient unconstrained minimization algorithms with adaptive stepsize are presented. The proposed class comprises four algorithms: the ÿrst two incorporate techniques for the adaptation of a common stepsize for all coordinate directions and the other two allow an individual adaptive stepsize along each coordinate direction. All the algorithms are computationally e cient and possess interesting convergence properties utilizing estimates of the Lipschitz constant that are obtained without additional function or gradient evaluations. The algorithms have been implemented and tested on some well-known test cases as well as on real-life artiÿcial neural network applications and the results have been very satisfactory.

Adaptive Algorithms for Neural Network Supervised Learning: A Deterministic Optimization Approach

“…Next we examine approaches to dynamically adapt the rate of learning that are based on optimization methods Magoulas et al, 1997aMagoulas et al, , 1997bVrahatis et al, 2000a;Anastasiadis et al, 2005a;Anastasiadis et al, 2005b]. In the context of unconstrained optimisation, Armijo's modified SD algorithm automatically adapts the rate of convergence [Armijo, 1966].…”

Modified Limited Memory BFGS Method With Nonmonotone Line Search for Unconstrained Optimization

“…where α k that is determined by a line search is the steplength, and d k which determines different line search methods [35,36,37,39,40,43,44,45,46,48,50] is a search direction of f at x k . One of the most effective methods for unconstrained optimization (1.1) is Newton method.…”

Convergence of descent method with new line search

“…Several choosing techniques have been appearing in the literature (e.g. [15,16,19,20,21]). The main motivation is to accelerate the convergence rate of related descent methods.…”