On the steplength selection in gradient methods for unconstrained optimization

Abstract

The seminal paper by Barzilai and Borwein (1988) has given rise to an extensive investigation, leading to the development of effective gradient methods. Several steplength rules have been first designed for unconstrained quadratic problems and then extended to general nonlinear optimization problems. These rules share the common idea of attempting to capture, in an inexpensive way, some second-order information. However, the convergence theory of the gradient methods using the previous rules does not explain their effectiveness, and a full understanding of their practical behaviour is still missing. In this work we investigate the relationships between the steplengths of a variety of gradient methods and the spectrum of the Hessian of the objective function, providing insight into the computational effectiveness of the methods, for both quadratic and general unconstrained optimization problems. Our study also identifies basic principles for designing effective gradient methods

Gradient methods exploiting spectral properties

“…From the total number of iterations, we can see the overall performance of the method (17) is quite good. Here, we want to point out that our method (17) and the DY method are monotone, while the ABB min 2 and SDC methods are not. Table 4 shows the averaged number of iterations of our method (18) for the first set of test problems.…”

“…Firstly, we compare our methods (17) and (18) with the DY method (6) in [14], the ABB min 2 method in [23], and the SDC method (7) in [16] for minimizing quadratic problems. Note that the SDC method has been shown performing better than its monotone variants [16].…”

“…Recently, the BB method and its variants have been successfully extended to general unconstrained problems [35], to constrained optimization problems [3,28] and to various applications [29,30,32]. One may see [4,10,17,21,37] and the references therein. Let {ξ 1 , ξ 2 , .…”

“…The recent study [17] points out that gradient methods using long and short stepsizes that attempt to exploit the spectral properties have generally better numerical performance for minimizing both quadratic and general nonlinear objective functions. For more works on gradient methods, see [7,9,16,17,23,25,37,39]. In [12], Dai and Yang introduced a gradient method with a new stepsize that possesses similar spectral property as α Y k .…”

On Extensions of Limited Memory Steepest Descent Method

“…Experience shows that computing Ritz values according to (4) can be a better choice in terms of stability at the expense of more arithmetic operations and extra long vectors. Fletcher [5] gave some remedies which consist of discarding the oldest back gradient vectors and recomputing matrix T .…”