Direct Method for Training Feed-Forward Neural Networks Using Batch Extended Kalman Filter for Multi-Step-Ahead Predictions

Abstract

This paper is dedicated to the long-term, or multi-step-ahead, time series prediction problem. We propose a novel method for training feed-forward neural networks, such as multilayer perceptrons, with tapped delay lines. Special batch calculation of derivatives called Forecasted Propagation Through Time and batch modification of the Extended Kalman Filter are introduced. Experiments were carried out on well-known timeseries benchmarks, the Mackey-Glass chaotic process and the Santa Fe Laser Data Series. Recurrent and feed-forward neural networks were evaluated.

Training Neural Networks for classification using the Extended Kalman Filter: A comparative study

“…Other computations are similar to (8)- (10). When B = 1, the mini batch train ing become sample by sample training.…”

“…IMPLEMENTATION OF MINI BATCH TRAINING IN THE EKF METHOD The simplest and most natural implementation of EKF based training is an on line sample by sample procedure where network weights are corrected for each new sample from the training dataset. However, with the EKF algorithms it is also possible to use mini batch training when a few samples are used for sin gle update of network weights [9,10]. In this case it can be regarded as correction of single neural net work's weights with K × B common outputs (where B is the number of training examples in a batch and K is the number of outputs of the actual network) or as correction of B "virtual" neural networks with shared weights.…”

Estimation of Vectors Similarity by Their Randomized Binary Projections

“…Distributed representations can also be formed in inner layers of multilayer networks in the course of training [46][47][48]. Distributed representations are used to represent semantic similarity [18,42,[49][50][51], sequences [29-31, 43, 51-54], complex hierarchically structured objects [1, 2, 4-11, 30, 31, 35, 36, 42-45, 55-57] required for models and systems of artificial intelligence [3,31,35,36,[57][58][59].…”