On spectral properties of steepest descent methods

Abstract

In recent years it has been made more and more clear that the critical issue in gradient methods is the choice of the step length, whereas using the gradient as search direction may lead to very effective algorithms, whose surprising behaviour has been only partially explained, mostly in terms of the spectrum of the Hessian matrix. On the other hand, the convergence of the classical Cauchy steepest descent (SD) method has been extensively analysed and related to the spectral properties of the Hessian matrix, but the connection with the spectrum of the Hessian has been little exploited to modify the method in order to improve its behaviour. In this work we show how, for convex quadratic problems, moving from some theoretical properties of the SD method, second-order information provided by the step length can be exploited to dramatically improve the usually poor practical behaviour of this method. This allows to achieve computational results comparable with those of the Barzilai and Borwein algorithm, with the further advantage of a monotonic behaviour.

On the steepest descent algorithm for quadratic functions

“…This leads to simple results that reproduce properties described by Nocedal, Sartenaer and Zhu [11] and by De Asmundis et al [6,7]. Consider an application of the Cauchy algorithm as above, starting from x 0 with no null component, and define r = lim k→∞ y 2k , with y k = g k / g k .…”

“…The reason for proving these results, which are not original, is that the original Akaike and Forsythe papers, not being aimed exclusively at these results, are frequently not considered easy to read. We also prove simplified versions of results in [6,7,11], with a unified notation.…”

“…In another line of research, several methods were developed to enhance the Cauchy algorithm by breaking its zig-zagging pattern. For the latest developments of this subject, see Asmundis et al [6,7]. This paper has two goals.…”

Geometrical inverse matrix approximation for least-squares problems and acceleration strategies

“…In general, the relationship between the choice of steplength, in gradient-type methods, and the eigenvalues and eigenvectors of the underlying Hessian of the objective function is well-known, and for nonmonotone methods can be traced back to [21, pp. 117-118]; see also [17,31].…”

A cyclic block coordinate descent method with generalized gradient projections

“…Usually, in iteration (2), λ (k) is adaptively computed to ensure the sufficient decrease of the objective function and, thus, the convergence of the whole scheme, while σ k is a 'free' parameter which can be chosen in order to improve the effectiveness of the algorithm (see e.g. [4,12,13,15]). In our analysis, we extend the convergence results about the gradient projection method (2)-(3) to the more general case where y (k) is defined as…”