Accelerating neural network training using weight extrapolations

Kamarthi, Sagar; Pittner, Stefan

doi:10.1016/s0893-6080(99)00072-6

Cited by 123 publications

(57 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Much of the existing analysis in the neural network literature (Haykin, 1999;Phansalkar & Sastry, 1994;Qian, 1999;Torii & Hagan 2002) is restricted to the case of constant l and m: There is also a large literature on the time-varying case, generally referred to as dynamic or adaptive choice of the learning rate and momentum factors (see Kamarthi and Pittner (1999) and references therein), but, to the best of our knowledge, the observations made in this paper are new. We now present the CG method from a control viewpoint, which is the inspiration for the results obtained here.…”

Section: Steepest Descent Plus Momentum Equals Frozen Conjugate Gradientmentioning

confidence: 96%

“…For a discussion of some of these practical computational issues, we refer the reader to Kamarthi and Pittner (1999) and Yu and Chen (1997) (and references therein).…”

Section: Practical Implications For Tuning Of Parametersmentioning

confidence: 99%

“…The backpropagation with momentum algorithm (BPM) has been much analyzed in the neural network literature and even compared with other methods, in particular, the conjugate gradient (CG) method (Kamarthi & Pittner, 1999;Yu & Chen, 1997). However, although it has been noticed in these two papers that BPM is actually a stationary (or time-invariant) version of the well known CG algorithm, this fact does not seem to have been enunciated explicitly enough in order to have permeated the neural network community, since papers continue to be written on the analysis of BPM without mentioning or exploiting this connection (Qian, 1999;Torii & Hagan, 2002).…”

Section: Introductionmentioning

confidence: 99%

“…For brevity, this note will focus on the contributions of Qian (1999) and Torii and Hagan (2002) which are recent and clearly written time-invariant analyses of the BPM method, which has been extensively analyzed, both theoretically and experimentally (see, for example, Hagiwara and Sato (1995), Kamarthi and Pittner (1999), Phansalkar and Sastry (1994), Yu and Chen (1997), and Yu, Chen, and Cheng (1995) and references therein).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Bhaya

Kaszkurewicz

2004

Neural Networks

View full text Add to dashboard Cite

It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system. q

show abstract

Section: Steepest Descent Plus Momentum Equals Frozen Conjugate Gradientmentioning

confidence: 96%

“…For a discussion of some of these practical computational issues, we refer the reader to Kamarthi and Pittner (1999) and Yu and Chen (1997) (and references therein).…”

Section: Practical Implications For Tuning Of Parametersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Bhaya

Kaszkurewicz

2004

Neural Networks

View full text Add to dashboard Cite

show abstract

“…The most popular learning algorithm used in prediction purposes is the back propagation algorithm (BPA) or the generalized delta rule. The BPA is a supervised learning algorithm that aims at reducing overall system error to a minimum [1,9]. This algorithm has made multilayer neural networks suitable for various prediction problems.…”

Section: Artificial Neural Network With Backpropagation Learning -An mentioning

confidence: 99%

Artificial neural network with backpropagation learning to predict mean monthly total ozone in Arosa, Switzerland

Chattopadhyay

Bandyopadhyay²

2007

International Journal of Remote Sensing

View full text Add to dashboard Cite

Present study deals with the mean monthly total ozone time series over Arosa, Switzerland. The study period is . First of all, the total ozone time series has been identified as a complex system and then Artificial Neural Networks models in the form of Multilayer Perceptron with back propagation learning have been developed. The models are Single-hidden-layer and Two-hidden-layer Perceptrons with sigmoid activation function. After sequential learning with learning rate 0.9 the peak total ozone period (February-May) concentrations of mean monthly total ozone have been predicted by the two neural net models. After training and validation, both of the models are found skillful. But, Two-hidden-layer Perceptron is found to be more adroit in predicting the mean monthly total ozone concentrations over the aforesaid period.

show abstract

Evolutionary Neural Networks

Siddique

Adeli

2013

Computational Intelligence

View full text Add to dashboard Cite

Accelerating neural network training using weight extrapolations

Cited by 123 publications

References 16 publications

Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Artificial neural network with backpropagation learning to predict mean monthly total ozone in Arosa, Switzerland

Evolutionary Neural Networks

Contact Info

Product

Resources

About