Supervised learning process of multi-layer perceptron neural networks using fast least squares

Azimi-Sadjadi, M.R.; Citrin, S.; Sheedvash, S.

doi:10.1109/icassp.1990.115644

Cited by 14 publications

(11 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For neural networks, however, the seriality property can not be assumed for the input and output sequences in each layer. Thus it is inevitable to use a RLS-based scheme [3], [4] for the weight adaptation procedure. It can be shown that the updating equation (28) is equivalent to that of the standard RLS by expressing the terms in Cj ( n)h N(n) and CN(n) using the RLS equations [13]; namely…”

Section: A Time Updating and Weight Adaptationmentioning

confidence: 99%

“…Equations (28), (31a), and (31b) represent the RLS algorithm developed in [3], [4] and CN(n) is the relevant gain matrix. Note that c~(n) is the a priori error (i.e., before updating) whereas C j (n) represents the a posteriori error (i.e., after updating).…”

Section: A Time Updating and Weight Adaptationmentioning

confidence: 99%

“…It is shown [4] that the problem of weight adjustment using the RLS approach leads to a normal equation for each layer. For example, for layer l we get the following normal equation:…”

Section: Introductionmentioning

confidence: 99%

“…Fast adaptation or learning is one of the main issues in these neural networks. In [3], [4], a new algorithm for expediting the learning process of multilayer neural networks is introduced using a recursive least squares (RLS) based method.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Recursive dynamic node creation in multilayer neural networks

Azimi-Sadjadi

Sheedvash

Trujillo

1993

IEEE Trans. Neural Netw.

Self Cite

View full text Add to dashboard Cite

configurations are tried, and if they do not yield an acceptable solution, they are discarded. Another topology is then defined and the whole training process is repeated. As a result, the possible benefits of training the original network architecture are lost and the computational costs of retraining become prohibitive. Another approach involves using a larger than needed topology and training it until a convergent solution is found. After that, the weights of the network are pruned off, if their values are negligible and have no influence on the performance of the network [7]. Since the pruning approach starts with a large network, the training time is larger than necessary and the method is computationally inefficient. It may also get trapped in one of the intermediately sized solutions because of the shape of the error surface and hence never finds the smallest network solution. Additionally, the relative importance of the nodes and weights depend on the particular mapping problem which the network is attempting to approximate and the pruning method makes it difficult to come up with a general cost function that would yield small networks for an arbitrary mapping. In the procedure suggested in [8], the error curve is monitored during the training process and a node is created when the ratio of the drop in the mean squared error (MSE) over a fixed number of trials falls below a priori chosen threshold slope. This procedure then uses the conventional, LMS-type, back-propagation algorithm to train the new architecture.In this paper a new recursive procedure for node creation in multilayer back-propagation neural networks is introduced. The derivations of the methodology are based upon the application of the Orthogonal Projection Theorem [12]. Simulation results on various examples are presented which indicate the effectiveness of the node creation scheme developed in this paper when used in conjunction with the RLS based learning method. II. TRAINING PROCESS OF MULTILAYER NEURAL NETWORKIn this section the problem of weight updating in multilayer neural networks is formulated in the context of the geometric orthogonal projection [11], [12]. The sum of the squared error is viewed as the squared length (or norm) of an error vector which is minimized using the geometric approach. It will be shown that the solution of the time updating leads to the RLS adaptation [9], [10], and the solution to the order updating allows us to recursively add nodes to the hidden layers during the training process.Consider an M-layer network as shown in Fig. 1 Abstract-This paper presents the derivations of a novel approach for simultaneous recursive weight adaptation and node creation in multilayer back-propagation neural networks. The method uses time and order update formulations in the orthogonal projection method to derive a recursive weight updating procedure for the training process of the neural network and a recursive node creation algorithm for weight adjustment of a layer with added nodes during the training process. The pr...

show abstract

Section: A Time Updating and Weight Adaptationmentioning

confidence: 99%

Section: A Time Updating and Weight Adaptationmentioning

confidence: 99%

“…It is shown [4] that the problem of weight adjustment using the RLS approach leads to a normal equation for each layer. For example, for layer l we get the following normal equation:…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Recursive dynamic node creation in multilayer neural networks

Azimi-Sadjadi

Sheedvash

Trujillo

1993

IEEE Trans. Neural Netw.

Self Cite

View full text Add to dashboard Cite

show abstract

“…(-) in all appropriate nodes, in principle any least square error algorithm can be used to update the weights. Algorithms based on similar ideas for updating weights one node at a time are given by Azimi-Sadjadi et al [5] (henceforth, A-S algorithm) and by Hunt and Deller [9]. The former is based on the conventional RLS algorithm [2] with a "If any weight connected a node is to be updated, then every weight connected to that node must be updated.…”

Section: Wvmentioning

confidence: 99%