Gradient Method with Retards and Generalizations

Abstract

A generalization of the steepest descent and other methods for solving a large scale symmetric positive definitive system Ax = b is presented. Given a positive integer m, the new iteration is given by x k+1 = x k − λ(x ν(k))(Ax k − b), where λ(x ν(k)) is the steepest descent step at a previous iteration ν(k) ∈ {k, k − 1,. .. , max{0, k − m}}. The global convergence to the solution of the problem is established under a more general framework, and numerical experiments are performed that suggest that some strategies for the choice of ν(k) give rise to efficient methods for obtaining approximate solutions of the system.

The cyclic Barzilai-–Borwein method for unconstrained optimization

“…., where α SD k is the exact steplength given by (1.3). Formula (1.4) was first proposed in Friedlander et al (1999), while the particular choice m = 2 was also investigated in Dai (2003) and Raydan & Svaiter (2002). The analysis in Dai & Fletcher (2005a) shows that if m > n 2 , cyclic SD is likely R-superlinearly convergent.…”

“…Other possible choices for the stepsize α k include Dai (2003), Dai & Fletcher (2006), Dai & Yang (2001, 2003, Friedlander et al (1999), Grippo & Sciandrone (2002), Raydan & Svaiter (2002) and Serafini et al (2005). In this paper, we refer to (1.6) as the BB formula.…”

“…In Friedlander et al (1999), the authors obtain global convergence of CBB when f is a strongly convex quadratic. Dai (2003) establishes the R-linear convergence of CBB for a strongly convex quadratic.…”

“…By the analysis given in Friedlander et al (1999) for positive definite quadratics or by the result given in Theorem 2.3 for general non-linear functions, the convergence rate of the iterates is at least linear. On the other hand, for m sufficiently large, we observe experimentally that the convergence rate is superlinear.…”

Cooperative concurrent asynchronous computation of the solution of symmetric linear systems

“…The BB algorithm has been much studied because of its remarkable improvement over the SD and OM algorithms ( [25,4,17] and references therein) and proofs of its convergence can be found in [20]. However, a complete explanation of why this simple modification of the SD algorithm improves its performance considerably has not yet been found, although it has been suggested that the improvement is connected to its nonmonotonic convergence, as well as to the fact that it does not produce iterates that get trapped in a low dimensional subspace [4].…”