Accelerated gradient descent methods with line search

Abstract

We introduced an algorithm for unconstrained optimization based on the transformation of the Newton method with the line search into a gradient descent method. Main idea used in the algorithm construction is approximation of the Hessian by an appropriate diagonal matrix. The steplength calculation algorithm is based on the Taylor's development in two successive iterative points and the backtracking line search procedure. The linear convergence of the algorithm is proved for uniformly convex functions and strictly convex quadratic functions satisfying specified conditions.

An Accelerated Double Step Size model in unconstrained optimization

“…There are many schemes with differently defined search direction vectors and step length parameters (see [4,[6][7][8]14,17,18,21,22]). Some of those ideas have directly influenced the development of the method described in this paper.…”

“…Some of those ideas have directly influenced the development of the method described in this paper. Certainly, one of the contributions that is of the great importance for this research is the algorithm presented in [21], stated in this paper as SM method. An interesting approach for calculating the accelerated parameter from [21] is applied on iteration defined in Section 3.…”

“…In [21], the authors named such a type of iterative schemes as accelerated gradient descent methods. Generally, mentioned acceleration parameter improves the behavior of gradient descent algorithms.…”

Implementation of gradient methods for optimization of underage costs in aviation industry

“…There are several iterative methods, each defined in a specific way, relevant for this work. Some of them are presented in articles (Andrei, 2006), (Stanimirović et al, 2010), (Petrović et al, 2014), , (Stanimirović et. al.…”

Machine Learning Techniques in Plant Biology

“…Speed of convergence and efficiency of gradient descent are improved using various variations like conjugate gradient descent (in which the weights are adjusted in directions conjugate to the gradient in order to get fast convergence) and gradient descent with adaptive (in which the learning rate is adjusted during the training in order to produce an optimum convergence rate and error) line search algorithms (Stanimirovic and Miladinovic 2010).…”