Recursive node creation in back-propagation neural networks using orthogonal projection method

Azimi-Sadjadi, M.R.; Sheedvash, S.

doi:10.1109/icassp.1991.150846

Cited by 5 publications

(5 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The order update equation and the node update formulation can be obtained by appending a column vector to the column space of un) [3]. When a new node N is added (see Figure 2), assuming that the weights to this node are randomly initialized, the corresponding vector is 7411) = [0, 0, ... , z4n)lZ = zdn) G, where zdn) is the output of the added node.…”

Section: Order Updating and Recursive Node Creationmentioning

confidence: 99%

See 1 more Smart Citation

A new approach for dynamic node creation in multilayer neural networks

Azimi-Sadjadi

Sheedvash

Trujillo

1991

[Proceedings] 1991 IEEE International Joint Conference on Neural Networks

Self Cite

View full text Add to dashboard Cite

This paper presents a novel approach for simultaneous recursive weight adaptation and node creation in multi-layer Perceptron neural networks. The method uses time and order update formulations in the orthogonal projection method to anive at a recursive weight updating procedure for training process of the neural network and a recursive node creation algorithm for weight adjustment of a layer with added nodes during the training process. The proposed approach allows optimal dynamic node creation in the Sense that the mean-squared error is minimized for each new topology. The effectiveness of the algorithm is demonstrated on a real world application for detecting and classifying underground dielectric anomalies.

show abstract

Section: Order Updating and Recursive Node Creationmentioning

confidence: 99%

“…The approach in [3] is used in this paper to arrive at a step by step procedure for recursive dynamic node creation during the RLS-based learning process. The simulation results on target detection and classification from microwave data are presented which indicate the effectiveness of the algorithm for real world applications.…”

Section: Introductionmentioning

confidence: 99%

A new approach for dynamic node creation in multilayer neural networks

Azimi-Sadjadi

Sheedvash

Trujillo

1991

[Proceedings] 1991 IEEE International Joint Conference on Neural Networks

Self Cite

View full text Add to dashboard Cite

show abstract

“…Azimi-Sadjadi and Sheedvash (1991) and Sin and de Figueiredo (1992) have used the RLS algorithm in training multilayer perceptrons (MLP). Other related work is the use of the extended Kalman filter (EKF) algorithm, which is similar in form to RLS, but allows one to incorporate knowledge or estimates of noise variances in the data.…”

Section: Introductionmentioning

confidence: 99%

A Dynamic Neural Network Architecture by Sequential Partitioning of the Input Space

Shadafan

Niranjan

1994

Neural Computation

View full text Add to dashboard Cite

We present a sequential approach to training multilayer perceptrons for pattern classification applications. The network is presented with each item of data only once and its architecture is dynamically adjusted during training. At the arrival of each example, a decision whether to increase the complexity of the network, or simply train the existing nodes is made based on three heuristic criteria. These criteria measure the position of the new item of data in the input space with respect to the information currently stored in the network. During the training process, each layer is assumed to be an independent entity with its particular input space. By adding nodes to each layer, the algorithm is effectively adding a hyperplane to the input space, hence adding a partition in the input space for that layer. When existing nodes are sufficient to accommodate the incoming input, the corresponding hidden nodes will be trained accordingly. Each hidden unit in the network is trained in closed form by means of a recursive least-squares (RLS) algorithm. A local covariance matrix of the data is maintained at each node and the closed form solution is recursively updated. The three criteria are computed from these covariance matrices to keep low computational cost. The performance of the algorithm is illustrated on two problems. The first problem is the two-dimensional Peterson and Barney vowel data. The second problem is a 33-dimensional data derived from a vision system for classifying wheat grains. The sequential nature of the algorithm has an efficient hardware implementation in the form of systolic arrays, and the incremental training idea has better biological plausibility compared with iterative methods.

show abstract

Section: Training Process Of Multilayer Neural Networkmentioning

confidence: 99%

“…The sum of the squared error is viewed as the squared length (or norm) of an error vector which is minimized using the geometric approach. It will be shown that the solution of the time updating leads to the RLS adaptation [9], [10], and the solution to the order updating allows us to recursively add nodes to the hidden layers during the training process.Consider an M-layer network as shown in Fig. 1 Abstract-This paper presents the derivations of a novel approach for simultaneous recursive weight adaptation and node creation in multilayer back-propagation neural networks.…”

mentioning

confidence: 99%

Recursive dynamic node creation in multilayer neural networks

Azimi-Sadjadi

Sheedvash

Trujillo

1993

IEEE Trans. Neural Netw.

Self Cite

View full text Add to dashboard Cite

configurations are tried, and if they do not yield an acceptable solution, they are discarded. Another topology is then defined and the whole training process is repeated. As a result, the possible benefits of training the original network architecture are lost and the computational costs of retraining become prohibitive. Another approach involves using a larger than needed topology and training it until a convergent solution is found. After that, the weights of the network are pruned off, if their values are negligible and have no influence on the performance of the network [7]. Since the pruning approach starts with a large network, the training time is larger than necessary and the method is computationally inefficient. It may also get trapped in one of the intermediately sized solutions because of the shape of the error surface and hence never finds the smallest network solution. Additionally, the relative importance of the nodes and weights depend on the particular mapping problem which the network is attempting to approximate and the pruning method makes it difficult to come up with a general cost function that would yield small networks for an arbitrary mapping. In the procedure suggested in [8], the error curve is monitored during the training process and a node is created when the ratio of the drop in the mean squared error (MSE) over a fixed number of trials falls below a priori chosen threshold slope. This procedure then uses the conventional, LMS-type, back-propagation algorithm to train the new architecture.In this paper a new recursive procedure for node creation in multilayer back-propagation neural networks is introduced. The derivations of the methodology are based upon the application of the Orthogonal Projection Theorem [12]. Simulation results on various examples are presented which indicate the effectiveness of the node creation scheme developed in this paper when used in conjunction with the RLS based learning method. II. TRAINING PROCESS OF MULTILAYER NEURAL NETWORKIn this section the problem of weight updating in multilayer neural networks is formulated in the context of the geometric orthogonal projection [11], [12]. The sum of the squared error is viewed as the squared length (or norm) of an error vector which is minimized using the geometric approach. It will be shown that the solution of the time updating leads to the RLS adaptation [9], [10], and the solution to the order updating allows us to recursively add nodes to the hidden layers during the training process.Consider an M-layer network as shown in Fig. 1 Abstract-This paper presents the derivations of a novel approach for simultaneous recursive weight adaptation and node creation in multilayer back-propagation neural networks. The method uses time and order update formulations in the orthogonal projection method to derive a recursive weight updating procedure for the training process of the neural network and a recursive node creation algorithm for weight adjustment of a layer with added nodes during the training process. The pr...

show abstract

Recursive node creation in back-propagation neural networks using orthogonal projection method

Cited by 5 publications

References 6 publications

A new approach for dynamic node creation in multilayer neural networks

A new approach for dynamic node creation in multilayer neural networks

A Dynamic Neural Network Architecture by Sequential Partitioning of the Input Space

Recursive dynamic node creation in multilayer neural networks

Contact Info

Product

Resources

About