Convergence Acceleration via Chebyshev Step: Plausible Interpretation of Deep-Unfolded Gradient Descent

Abstract

Deep unfolding is a promising deep-learning technique, whose network architecture is based on expanding the recursive structure of existing iterative algorithms. Although convergence acceleration is a remarkable advantage of deep unfolding, its theoretical aspects have not been revealed yet. The first half of this study details the theoretical analysis of the convergence acceleration in deep-unfolded gradient descent (DUGD) whose trainable parameters are step sizes. We propose a plausible interpretation of the learned step-size parameters in DUGD by introducing the principle of Chebyshev steps derived from Chebyshev polynomials. The use of Chebyshev steps in gradient descent (GD) enables us to bound the spectral radius of a matrix governing the convergence speed of GD, leading to a tight upper bound on the convergence rate. The convergence rate of GD using Chebyshev steps is shown to be asymptotically optimal, although it has no momentum terms. We also show that Chebyshev steps numerically explain the learned step-size parameters in DUGD well. In the second half of the study, Chebyshev-periodical successive over-relaxation (Chebyshev-PSOR) is proposed for accelerating linear/nonlinear fixed-point iterations. Theoretical analysis and numerical experiments indicate that Chebyshev-PSOR exhibits significantly faster convergence for various examples such as Jacobi method and proximal gradient methods.

Acceleration of Cooperative Least Mean Square via Chebyshev Periodical Successive Over-Relaxation

“…In this subsection, we will briefly review some basic facts regarding Chebyshev PSOR according to [9].…”

“…as an example. The method in [9] handles more general fixed-point iterations, such as x (k+1) := f (x (k) ) but we here restrict our attention to the simplest case required for the following discussion. If the spectral radius of A satisfies ρ(A) < 1, the linear fixed-point iteration converges to the fixed point x * = 0.…”

“…) lead to faster convergence. The paper [9] introduced an affine translate of the Chebyshev polynomial having the desired properties. We define the Chebyshev PSOR factors {ω ch k } T −1 k=0 for the range [a, b] by…”

“…The authors proposed a method to accelerate the convergence of a fixed-point iteration in [9]. The acceleration method is called Chebyshev periodical successive over-relaxation (Chbyshev PSOR), and it is applicable both to linear and nonlinear fixed-point iterations.…”

“…The name of the method is named after the Chebyshev polynomials that are used for determining the PSOR factors. In [9], it is shown that many fixed-point iterations, such as the Jacobi method for solving linear equations are successfully accelerated.…”