A novel quasi-Newton-based optimization for neural network training incorporating Nesterov's accelerated gradient

Abstract

This paper describes a novel quasi-Newton (QN) based accelerated technique for training of neural networks. Recently, Nesterov's accelerated gradient method has been utilized for the acceleration of the gradient-based training. In this paper the acceleration of the QN training algorithm is realized by the quadratic approximation of the error function incorporating the momentum term as Nesterov's method. It is shown that the proposed algorithm has a similar convergence property with the QN method. Neural network trainings for the function approximation and the microwave circuit modeling problems are presented to demonstrate the proposed algorithm. The method proposed here drastically improves the convergence speed of the conventional QN algorithm.

A Robust Quasi-Newton Training with Adaptive Momentum for Microwave Circuit Models in Neural Networks

“…In this section, NAQ [9][13] and the adaptive momentum coefficient scheme [11] is introduced. The proposed AdaNAQ algorithm combines NAQ with the adaptive momentum coefficient scheme [14].…”

“…The former [5][6] [7] [8] are often used and improved because of the simplicity of their calculation. However, when applied to highly nonlinear problems, first-order methods still converge too slowly and the optimization error cannot be effectively reduced within a finite time [1] [9]. To deal with this problem, the quasi-Newton method (QN), which is one of the most efficient optimization methods with superlinear convergence has been widely utilized as a robust training algorithm [10].…”

“…To deal with this problem, the quasi-Newton method (QN), which is one of the most efficient optimization methods with superlinear convergence has been widely utilized as a robust training algorithm [10]. Furthermore, the QN was accelerated by the momentum term as in Nesterov's accelerated quasi-Newton method (NAQ) in [9]. However, the momentum coefficient in most NN training is a problem-dependent parameter, that is, a hyperparameter.…”

Implementation of a Modified Nesterov's Accelerated Quasi-Newton Method on Tensorflow

“…Several modifications have been proposed to quasi-Newton to obtain stronger convergence. The Nesterov's Accelerated quasi-Newton method [15] gives faster convergence compared to the standard quasi-Newton methods. NAQ obtains faster convergence by quadratic approximation at w k + µv k and by incorporating the Nesterov's accelerated gradient ∇E(w k + µv k ) The derivation of NAQ is briefly introduced as follows:…”

A Stochastic Variance Reduced Nesterov's Accelerated Quasi-Newton Method

“…The NAQ [16] method achieves faster convergence compared to the standard QN by quadratic approximation of the objective function at w k + µv k and by incorporating the Nesterov's accelerated gradient ∇E(w k + µv k ) in its Hessian update. The update vector of NAQ is given as…”