On the Kalman filtering method in neural network training and pruning

Abstract

Abstract-In the use of extended Kalman filter approach in training and pruning a feedforward neural network, one usually encounters the problems on how to set the initial condition and how to use the result obtained to prune a neural network. In this paper, some cues on the setting of the initial condition will be presented with a simple example illustrated. Then based on three assumptions-1) the size of training set is large enough; 2) the training is able to converge; and 3) the trained network model is close to the actual one, an elegant equation linking the error sensitivity measure (the saliency) and the result obtained via extended Kalman filter is devised. The validity of the devised equation is then testified by a simulated example.Index Terms-Extended Kalman filter, multilayer perceptron, pruning training, weight saliency.

RBF networks training using a dual extended Kalman filter

“…As will be pointed out in the next paragraph, this idea is also useful for assessing the relative importance (saliency) of the network parameters, which may be pruned accordingly [17].…”

“…(7). Denoting P ∞ = lim k→∞ P[k] and the covariance matrix of the process noise Q = qI; the incremental change of the approximation error due to removing the kth element of the (parameters) state-vector W is given by [17] …”

“…Eliminating eigennodes should improve generalization performances by reducing the number of e ective number of parameters of the network [25]. (b) Recently, a number of papers have addressed pruning within a Kalman ÿltering framework [17,26]. The relative importance (saliency) of the parameters of a feedforward neural network is assessed using speciÿc information available after the training period, namely the ÿnal values of the error covariance matrix P from Eq.…”

Metaheuristic design of feedforward neural networks: A review of two decades of research

“…Many researchers suggested that the Levenberg-Marquardt (LM) method outperforms BP, CG, and Quasi-Newton methods [79,80]. Several other methods were proposed for the FNNs optimization are based on Kalman-filter [81,82] and recursive least squares method [83].…”

Training RBF Networks with an Extended Kalman Filter Optimized Using Fuzzy Logic

“…Various derivative-based methods have been used to train neural networks, including gradient descent [3], Kalman filtering [4,5], and backpropagation [9], etc. Gradient descent training of RBF networks has proven to be much more effective than more conventional methods [3].However; gradient descent training can be computationally expensive.…”