Convergence of the Momentum Method for Semialgebraic Functions with Locally Lipschitz Gradients

Josz, Cédric; Lai, Lexiao; Li, Xiaopeng

doi:10.1137/23m1545720

Cited by 2 publications

(2 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Positive values of φ ′ 1 (λ) can be realized when the following inequality is valid: (15). Let us demonstrate that (16) correlates with (14): Inequity (16)…”

Section: And In This Casementioning

confidence: 94%

“…In order to carry out the damping of this effect in [14], an averaged HBM was constructed. A global and local convergence of momentum method for semialgebraic functions with locally Lipschitz gradients was demonstrated in [15]. Wang et al [16] used the theory of PID controllers for the construction of momentum methods for deep neural network training.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of a Two-Step Gradient Method with Two Momentum Parameters for Strongly Convex Unconstrained Optimization

Krivovichev,

Sergeeva

2024

Algorithms

View full text Add to dashboard Cite

The paper is devoted to the theoretical and numerical analysis of the two-step method, constructed as a modification of Polyak’s heavy ball method with the inclusion of an additional momentum parameter. For the quadratic case, the convergence conditions are obtained with the use of the first Lyapunov method. For the non-quadratic case, sufficiently smooth strongly convex functions are obtained, and these conditions guarantee local convergence.An approach to finding optimal parameter values based on the solution of a constrained optimization problem is proposed. The effect of an additional parameter on the convergence rate is analyzed. With the use of an ordinary differential equation, equivalent to the method, the damping effect of this parameter on the oscillations, which is typical for the non-monotonic convergence of the heavy ball method, is demonstrated. In different numerical examples for non-quadratic convex and non-convex test functions and machine learning problems (regularized smoothed elastic net regression, logistic regression, and recurrent neural network training), the positive influence of an additional parameter value on the convergence process is demonstrated.

show abstract

“…Positive values of φ ′ 1 (λ) can be realized when the following inequality is valid: (15). Let us demonstrate that (16) correlates with (14): Inequity (16)…”

Section: And In This Casementioning

confidence: 94%