Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

Abstract

In this paper, we propose the greedy and random Broyden's method for solving nonlinear equations. Specifically, the greedy method greedily selects the direction to maximize a certain measure of progress for approximating the current Jacobian matrix, while the random method randomly chooses a direction. We establish explicit (local) superlinear convergence rates of both methods if the initial point and approximate Jacobian are close enough to a solution and corresponding Jacobian. Our two novel variants of Broyden's method enjoy two important advantages that the approximate Jacobians of our algorithms will converge to the exact ones and the convergence rates of our algorithms are asymptotically faster than the original Broyden's method. Our work is the first time to achieve such two advantages theoretically. Our experiments also empirically validate the advantages of our algorithms.

Quasi-Newton Methods for Saddle Point Problems and Beyond

“…Related Work Very recently, Lin et al [19], Ye et al [38] showed Broyden's methods have explicit local superlinear convergence rate for solving nonlinear equations, but their assumptions are quite different from ours. Specifically, they only suppose Ĥ(z) − Ĥ(z * ) ≤ L 2 z − z * for any z ∈ R n , which is weaker than Assumption 4.1 of this paper.…”

“…Similarly, we can also achieve G k+1 by such Gk for specific BFGS and SR1 update. Combining iteration (10) with (38), we propose the quasi-Newton methods for general strongly-convex-strongly-concave saddle point problems. The details is shown in Algorithm 5, 6 and 7 for greedy Broyden family, BFGS and SR1 updates respectively.…”

Smooth monotone stochastic variational inequalities and saddle point problems: A survey

“…Quasi-Newton methods for solving nonlinear equations (unconstrained VI) and SPP are proposed in [75,121] and [79], respectively. In these papers, local superlinear rates of convergence are derived for the modifications of the Broyden-type methods for solving nonlinear equations with Lipschitz Jacobian and SPP with Lipschitz Hessian.…”