Gradient methods exploiting spectral properties

Abstract

We propose a new stepsize for the gradient method. It is shown that this new stepsize will converge to the reciprocal of the largest eigenvalue of the Hessian, when Dai-Yang's asymptotic optimal gradient method (Computational Optimization and Applications, 2006, 33(1): 73-88) is applied for minimizing quadratic objective functions. Based on this spectral property, we develop a monotone gradient method that takes a certain number of steps using the asymptotically optimal stepsize by Dai and Yang, and then follows by some short steps associated with this new stepsize. By employing one step retard of the asymptotic optimal stepsize, a nonmonotone variant of this method is also proposed. Under mild conditions, R-linear convergence of the proposed methods is established for minimizing quadratic functions. In addition, by combining gradient projection techniques and adaptive nonmonotone line search, we further extend those methods for general bound constrained optimization. Two variants of gradient projection methods combining with the Barzilai-Borwein stepsizes are also proposed. Our numerical experiments on both quadratic and bound constrained optimization indicate that the new proposed strategies and methods are very effective.

On the Asymptotic Convergence and Acceleration of Gradient Methods

“…However, these methods are based on the SD method, i.e., occasionally applying short steps during the iterates of the SD method. One exception is given by [21], where a method is developed by employing new stepsizes during the iterates of the AOPT method. But our method periodically uses three different stepsizes: the BB stepsize, stepsize (11) and the new stepsize αk .…”

“…We have tested five different sets of problems in the form of (92) with spectrum distributions described in Table 2, see [21,30] for example. The rand function in Matlab was used for generating Hessian of the first three problem sets.…”

“…However, the AOPT method also asymptotically alternates between two directions. To accelerate the AOPT method, Huang et al [21] derived a new stepsize that converges to 1/λ n during the AOPT iterates and further suggested a gradient method to exploit spectral properties of the stepsizes. For the latest developments of exploiting spectral properties to accelerate gradient methods, see [12][13][14]20,21] and references therein.…”

A new family of Dai-Liao conjugate gradient methods with modified secant equation for unconstrained optimization

“…numerical solution methods have been intensively studied in the literature, including the spectral gradient methods [5,15], conjugate gradient methods [4,13] and memoryless BFGS methods [16]. Among them, conjugate gradient methods are popular and efficient for solving (1), especially for large scale problems.…”

Variable metric proximal stochastic variance reduced gradient methods for nonconvex nonsmooth optimization

“…Yu et al [35] combined the trust-region scheme and BB stepsizes with SARAH for solving nonsmooth convex composite problems. A remarkable advantage of the stepsize given by the BB method is that it estimates a scalar approximation of the Hessian and is not sensitive to the choice of initial stepsizes, see [5,7,11] and references therein for more details about BB-like methods. However, the research on incorporating BB stepsizes with proximal stochastic gradient methods in the nonconvex nonsmooth case is far less than in the convex case.…”