Technique of learning rate estimation for efficient training of MLP

Golovko, Vladimir; Savitsky, Yury; Laopoulos, T.; Sachenko, Anatoly; Grandinetti, Lucio

doi:10.1109/ijcnn.2000.857856

Cited by 24 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…is used for the neurons of the hidden ( 2 F ) and output layer ( 3 F ) for the both MLP and RNN models. The standard back-propagation training algorithm [11] with constant or adaptive learning rate [20] is used for the training for both NN models.…”

Section: Neural-based Prediction Methodsmentioning

confidence: 99%

Spot Price Prediction for Cloud Computingusing Neural Networks

Turchenko¹,

Shults²,

Turchenko³

et al. 2014

IJC

View full text Add to dashboard Cite

Advances in service-oriented architectures, virtualization, high-speed networks, and cloud computing has resulted in attractive pay-as-you-go services. Job scheduling on such systems results in commodity bidding for computing time. Amazon institutionalizes this bidding for its Elastic Cloud Computing (EC2) environment. Similar bidding methods exist for other cloud-computing vendors as well as multi–cloud and cluster computing brokers such as SpotCloud. Commodity bidding for computing has resulted in complex spot price models that have ad-hoc strategies to provide demand for excess capacity. In this paper we will discuss vendors who provide spot pricing and bidding and present the predictive models for future short-term and middle-term spot price prediction based on neural networks giving users a high confidence on future prices aiding bidding on commodity computing.

show abstract

Section: Neural-based Prediction Methodsmentioning

confidence: 99%

Spot Price Prediction for Cloud Computingusing Neural Networks

Turchenko¹,

Shults²,

Turchenko³

et al. 2014

IJC

View full text Add to dashboard Cite

show abstract

“…Such learning rate can only be obtained for linear and ReLU activation functions. When using the sigmoid activation function, we can only receive approximate expressions for the learning rate using the Taylor series expansion ( Golovko et al, 2000 ; Golovko, 2003 ). Since this is a very complicated problem, as mentioned before, most of the scientists use the steepest descent method together with the line search approach.…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, in this paper we investigate the calculation of adaptive learning rate for a RBM, which is based on the steepest descent technique ( Golovko et al, 2000 , 2023 ; Golovko, 2003 ). This approach is based on minimizing the loss function to calculate the adaptive learning step.…”

Section: Introductionmentioning

confidence: 99%

An analytical approach for unsupervised learning rate estimation using rectified linear units

Chen,

Golovko,

Kroshchanka

et al. 2024

Front. Neurosci.

View full text Add to dashboard Cite

Unsupervised learning based on restricted Boltzmann machine or autoencoders has become an important research domain in the area of neural networks. In this paper mathematical expressions to adaptive learning step calculation for RBM with ReLU transfer function are proposed. As a result, we can automatically estimate the step size that minimizes the loss function of the neural network and correspondingly update the learning step in every iteration. We give a theoretical justification for the proposed adaptive learning rate approach, which is based on the steepest descent method. The proposed technique for adaptive learning rate estimation is compared with the existing constant step and Adam methods in terms of generalization ability and loss function. We demonstrate that the proposed approach provides better performance.

show abstract

“…The steepest descent method for calculating the adaptive learning rate [4] is used for removing the classical disadvantages of the back propagation error algorithm with empirical choosing of the learning rate.…”

Section: Multi-layer Perceptron Modelmentioning

confidence: 99%

Interplanetary Shock Arrival Time Prediction Using Multi-Layer Perceptron

Turchenko

Demchuk

Sachenko

2007

2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications

View full text Add to dashboard Cite

An approach to prediction of the arrival time of interplanetary shocks using neural networks based on the data gathered from EPAM (Electron, Proton and Alpha Monitor) sensor of NASA's ACE (Advanced Composition Explorer) spacecraft is proposed in this paper. A short description of ACE spacecraft and the data, published online on the appropriate web-site, are presented. A data preparation for prediction is proven and multi-layer perceptron architecture is described. The results of simulation modelling are presented in the end of the paper.

show abstract

Technique of learning rate estimation for efficient training of MLP

Cited by 24 publications

References 9 publications

Spot Price Prediction for Cloud Computingusing Neural Networks

Spot Price Prediction for Cloud Computingusing Neural Networks

An analytical approach for unsupervised learning rate estimation using rectified linear units

Interplanetary Shock Arrival Time Prediction Using Multi-Layer Perceptron

Contact Info

Product

Resources

About