Matrix algebras in Quasi-Newton methods for unconstrained minimization

Abstract

Summary. In this paper a new class of quasi-Newton methods, named LQN, is introduced in order to solve unconstrained minimization problems. The novel approach, which generalizes classical BF GS methods, is based on a Hessian updating formula involving an algebra L of matrices simultaneously diagonalized by a fast unitary transform. The complexity per step of LQN methods is O(n log n), thereby improving considerably BF GS computational efficiency. Moreover, since LQN's iterative scheme utilizes single-indexed arrays, only O(n) memory allocations are required. Global convergence properties are investigated. In particular a global convergence result is obtained under suitable assumptions on f . Numerical experiences [7] confirm that LQN methods are particularly recommended for large scale problems.

Adaptive matrix algebras in unconstrained minimization

“…For a more exhaustive treatment of the contents of Lemma 2 and Lemma 3, and their relevance for LQN minimizations algorithms and optimal preconditioning, one can see [11] and [7].…”

“…In particular, if such condition holds, then (2) is obtained by choosing B k+1 in a suitable algebra L (k+1) = sd U k+1 where U k+1 is defined as the product of two Householder unitary matrices, and the effect of this choice is to reduce to O(n) both computational cost per step and memory allocations. Moreover, the L (k) QN methods obtained by forcing at each step equality (2) turn out to be globally convergent, and this is not surprising since the sequence of approximations {x k } k∈N they yield can be seen as produced by the corresponding Non-Secant L (k) QN methods defined in terms of the search directions −B −1 k+1 g k+1 (it is known that Non-Secant L (k) QN methods are globally convergent [7,8]). …”

“…In [7,10,8,9,6] it was introduced a new class of algorithms, named LQN, which includes methods of Secant type, in particular the well known BFGS method, and, at the same time, some methods which are not Secant but have relevant good properties (f.i. global convergence).…”

“…L = sd U = {L = Ud(z)U H } where d(z) denotes the diagonal matrix of the eigenvalues z i of L. In fact, if B k is the best approximation L B k in L of B k in Frobenius norm, then from (1) we obtain a simple single-array iteration to compute the eigenvalues of L B k+1 from the eigenvalues of L B k [7]. At least two choices are possible for the new descent direction d k+1 :…”

“…Moreover, a global convergent result has been obtained for NSLQN algorithms [7]. However, one may expect that their secant version SLQN, with more cumbersome formulas but roughly with the same cost per step, are more efficient than NSLQN, even if no proof of convergence of SLQN has been found.…”