Momentum acceleration of quasi-Newton based optimization technique for neural network training

Abstract

This paper describes a momentum acceleration technique for quasi-Newton (QN) based neural network training and verifies its performance and computational complexity. Recently, Nesterov's accelerated quasi-Newton method (NAQ) has been introduced and shown that the momentum term is effective in reducing the number of iterations and the total training time by incorporating Nesterov's accelerated gradient into QN. However, the gradients had to be calculated two times in one iteration in the NAQ training. This increased the computation time of a training loop compared with the conventional QN. The proposed technique is an improvement to NAQ done by approximating the Nesterov's accelerated gradient as a linear combination of the current and previous gradients. As a result, the gradient is calculated only once per iteration similar to that of QN. The performance of the proposed algorithm is evaluated in comparison to conventional algorithms in neural networks training on two types of problems -function approximation problems with high nonlinearity and classification problems. The results show a significant acceleration in the computation time without losing the quality of the solution compared with conventional training algorithms.

Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov’s Gradient for Training Neural Networks

“…Also note that the proposed L-SR1-N has two gradient computations per iteration. The Nesterov's gradient ∇E(w k + µ k v k ) can be approximated [25,29] as a linear combination of past gradients as shown below.…”

“…Recent works such as [22,23] have proposed sampled LSR1 (limited memory) quasi-Newton updates for machine learning and describe efficient ways for distributed training implementation. Recent studies such as [24,25] have shown that the BFGS method can be accelerated by using Nesterov's accelerated gradient and momentum terms. In this paper, we explore if the Nesterov's acceleration can be applied to the LSR1 quasi-Newton method as well.…”

Battery Sizing Optimization in Power Smoothing Applications

“…Often, these algorithms reach, yet do not leave, local minima points. Their convergence decreases heavily when they reach certain local optima, although momentum techniques are very common [14]. In this gradient descent algorithm, explicit mathematical expressions are needed in order to obtain the gradient components.…”

On the Practical Robustness of the Nesterov’s Accelerated Quasi-Newton Method

“…(8) where µ is the momentum parameter and ∇f (θ k + µv k ) is the Nesterov's accelerated gradient. MoQ (Mahboubi et al 2021) approximated ∇f (θ k +µv k ) in NAQ as a linear combination of past gradients. The acceleration of second order methods pave promising scope to numerous applications and is the focus of this research.…”