Recursive dynamic node creation in multilayer neural networks

Abstract

configurations are tried, and if they do not yield an acceptable solution, they are discarded. Another topology is then defined and the whole training process is repeated. As a result, the possible benefits of training the original network architecture are lost and the computational costs of retraining become prohibitive. Another approach involves using a larger than needed topology and training it until a convergent solution is found. After that, the weights of the network are pruned off, if their values are negligible and have no influence on the performance of the network [7]. Since the pruning approach starts with a large network, the training time is larger than necessary and the method is computationally inefficient. It may also get trapped in one of the intermediately sized solutions because of the shape of the error surface and hence never finds the smallest network solution. Additionally, the relative importance of the nodes and weights depend on the particular mapping problem which the network is attempting to approximate and the pruning method makes it difficult to come up with a general cost function that would yield small networks for an arbitrary mapping. In the procedure suggested in [8], the error curve is monitored during the training process and a node is created when the ratio of the drop in the mean squared error (MSE) over a fixed number of trials falls below a priori chosen threshold slope. This procedure then uses the conventional, LMS-type, back-propagation algorithm to train the new architecture.In this paper a new recursive procedure for node creation in multilayer back-propagation neural networks is introduced. The derivations of the methodology are based upon the application of the Orthogonal Projection Theorem [12]. Simulation results on various examples are presented which indicate the effectiveness of the node creation scheme developed in this paper when used in conjunction with the RLS based learning method. II. TRAINING PROCESS OF MULTILAYER NEURAL NETWORKIn this section the problem of weight updating in multilayer neural networks is formulated in the context of the geometric orthogonal projection [11], [12]. The sum of the squared error is viewed as the squared length (or norm) of an error vector which is minimized using the geometric approach. It will be shown that the solution of the time updating leads to the RLS adaptation [9], [10], and the solution to the order updating allows us to recursively add nodes to the hidden layers during the training process.Consider an M-layer network as shown in Fig. 1 Abstract-This paper presents the derivations of a novel approach for simultaneous recursive weight adaptation and node creation in multilayer back-propagation neural networks. The method uses time and order update formulations in the orthogonal projection method to derive a recursive weight updating procedure for the training process of the neural network and a recursive node creation algorithm for weight adjustment of a layer with added nodes during the training process. The pr...

SUSTAIN: A Network Model of Category Learning.

“…Indeed, some learning models have an adaptive architecture and adopt this approach. For instance, some models begin large and reduce unneeded complexity (Karnin, 1990;Busemeyer & McDaniel, 1997), whereas other adaptive architecture models (including SUS-TAIN) begin small and expand as needed (Ash, 1989;Carpenter, Grossberg, & Reynolds, 1991;Cho, 1997;Fahlman & Lebiere, 1990;Kruschke & Movellan, 1991;Azimi-Sadjadi, Sheedvash, & Trujillo, 1993) Adaptive architecture learning models can be quite effective in mastering a wide range of learning problems because they can adapt their complexity to the current problem. Humans face a similar challenge.…”

An evolutionary algorithm that constructs recurrent neural networks

“…Current methods to solve this task fall into two broad categories. Constructive algorithms initially assume a simple network and add nodes and links as warranted [2][3][4][5][6][7][8], while destructive methods start with a large network and prune off superfluous components [9][10][11][12]. Though these algorithms address the problem of topology acquisition, they do so in a highly constrained manner.…”

Enhancements to the dynamic self-organising neural network

“…Although dynamic network architectures have in the past been limited to supervised paradigms [24][25][26][27][28][29], DLVQ recently provided a means of adding nodes by utilising information fed to the network by an external signal. CDUL however, provides an independent dynamic unsupervised network architecture, by using an integral class representation, balanced against computed classifications, to act as a basis upon which new nodes may be generated.…”