Momentum Acceleration of Quasi-Newton Training for Neural Networks

Mahboubi, Shahrzad; Indrapriyadarsini, S.; Ninomiya, Hiroshi; Asai, Hideki

doi:10.1007/978-3-030-29911-8_21

Cited by 5 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this research, a new quasi-Newton based training algorithm -Momentum quasi-Newton method (MoQ) -is introduced showing the performance for not only the function approximation problems but also the classification ones. Actually, our MoQ was proposed and its effectiveness for only the function approximation problems was confirmed in [16]. This paper extends the validity of MoQ for neural network training using classification problems.…”

Section: Introductionsupporting

confidence: 53%

“…From ( 14) and ( 15), it is confirmed that Nesterov's accelerated gradient can be approximated as an extrapolation of ∇E(w k ) and ∇E(w k−1 ) with a momentum coefficient μ k , that is, a weighted linear combination [16]. In this paper,…”

Section: Momentum Quasi-newton Methods (Moq)mentioning

confidence: 59%

“…While QN calculates only the normal gradient per iteration of the training loop, NAQ involves two gradient calculations -Nesterov's accelerated gradient ∇E(w k + μ k v k ) and the normal gradient ∇E(w k+1 ). In order to reduce the number of gradient computations per iteration without losing the performance of NAQ, we proposed a novel quasi-Newton based training algorithm called Momentum quasi-Newton method (MoQ) [16]. In this paper, we further test the limits of our proposed method by evaluating its performance on a variety of problems and analyze its computational cost.…”

Section: Proposed Algorithm -Momentum Quasi-newton Methods (Moq)mentioning

confidence: 99%

“…Furthermore, in order to guarantee the numerical stability and the global convergence of MoQ, a global convergence term is incorporated inq k , as shown in ( 26), ( 27) and ( 28) [22]. In this paper, this term is applied to QN [22], MSQN [16], NAQ [23] and MoQ for fair comparison.…”

Section: Proof Of (C)mentioning

confidence: 99%

“…However, when applied to function approximation problems with high nonlinearity, these first-order methods still converge too slowly and optimization error cannot be effectively reduced within finite time in spite of its advantage [14]. Typically, second-order methods have been used to deal with this problem despite the increase in computational cost [1,[14][15][16][17][18][19] caused by the curvature information of the error function like Hessian or the approximated Hessian computation. In second-order methods, η k of (3) is determined using a scalar α k and a matrix H k as α k H k unlike first-order methods that used a scaler.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Momentum acceleration of quasi-Newton based optimization technique for neural network training

Mahboubi

Indrapriyadarsini

Ninomiya

et al. 2021

NOLTA

Self Cite

View full text Add to dashboard Cite

This paper describes a momentum acceleration technique for quasi-Newton (QN) based neural network training and verifies its performance and computational complexity. Recently, Nesterov's accelerated quasi-Newton method (NAQ) has been introduced and shown that the momentum term is effective in reducing the number of iterations and the total training time by incorporating Nesterov's accelerated gradient into QN. However, the gradients had to be calculated two times in one iteration in the NAQ training. This increased the computation time of a training loop compared with the conventional QN. The proposed technique is an improvement to NAQ done by approximating the Nesterov's accelerated gradient as a linear combination of the current and previous gradients. As a result, the gradient is calculated only once per iteration similar to that of QN. The performance of the proposed algorithm is evaluated in comparison to conventional algorithms in neural networks training on two types of problems -function approximation problems with high nonlinearity and classification problems. The results show a significant acceleration in the computation time without losing the quality of the solution compared with conventional training algorithms.

show abstract

Section: Introductionsupporting

confidence: 53%

Section: Momentum Quasi-newton Methods (Moq)mentioning

confidence: 59%

Section: Proposed Algorithm -Momentum Quasi-newton Methods (Moq)mentioning

confidence: 99%

Section: Proof Of (C)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Momentum acceleration of quasi-Newton based optimization technique for neural network training

Mahboubi

Indrapriyadarsini

Ninomiya

et al. 2021

NOLTA

Self Cite

View full text Add to dashboard Cite

show abstract

Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov’s Gradient for Training Neural Networks

Indrapriyadarsini¹,

Mahboubi²,

Ninomiya³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Gradient based methods are popularly used in training neural networks and can be broadly categorized into first and second order methods. Second order methods have shown to have better convergence compared to first order methods, especially in solving highly nonlinear problems. The BFGS quasi-Newton method is the most commonly studied second order method for neural network training. Recent methods have shown to speed up the convergence of the BFGS method using the Nesterov’s acclerated gradient and momentum terms. The SR1 quasi-Newton method though less commonly used in training neural networks, are known to have interesting properties and provide good Hessian approximations when used with a trust-region approach. Thus, this paper aims to investigate accelerating the Symmetric Rank-1 (SR1) quasi-Newton method with the Nesterov’s gradient for training neural networks and briefly discuss its convergence. The performance of the proposed method is evaluated on a function approximation and image classification problem.

show abstract

Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov’s Gradient for Training Neural Networks

et al. 2021

Self Cite

View full text Add to dashboard Cite

Gradient-based methods are popularly used in training neural networks and can be broadly categorized into first and second order methods. Second order methods have shown to have better convergence compared to first order methods, especially in solving highly nonlinear problems. The BFGS quasi-Newton method is the most commonly studied second order method for neural network training. Recent methods have been shown to speed up the convergence of the BFGS method using the Nesterov’s acclerated gradient and momentum terms. The SR1 quasi-Newton method, though less commonly used in training neural networks, is known to have interesting properties and provide good Hessian approximations when used with a trust-region approach. Thus, this paper aims to investigate accelerating the Symmetric Rank-1 (SR1) quasi-Newton method with the Nesterov’s gradient for training neural networks, and to briefly discuss its convergence. The performance of the proposed method is evaluated on a function approximation and image classification problem.

show abstract

Momentum Acceleration of Quasi-Newton Training for Neural Networks

Cited by 5 publications

References 12 publications

Momentum acceleration of quasi-Newton based optimization technique for neural network training

Momentum acceleration of quasi-Newton based optimization technique for neural network training

Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov’s Gradient for Training Neural Networks

Accelerating Symmetric Rank-1 Quasi-Newton Method with Nesterov’s Gradient for Training Neural Networks

Contact Info

Product

Resources

About