Enhancing MLP networks using a distributed data representation

Abstract

Multilayer perceptron (MLP) networks trained using backpropagation can be slow to converge in many instances. The primary reason for slow learning is the global nature of backpropagation. Another reason is the fact that a neuron in an MLP network functions as a hyperplane separator and is therefore inefficient when applied to classification problems in which decision boundaries are nonlinear. This paper presents a data representational approach that addresses these problems while operating within the framework of the familiar backpropagation model. We examine the use of receptors with overlapping receptive fields as a preprocessing technique for encoding inputs to MLP networks. The proposed data representation scheme, termed ensemble encoding, is shown to promote local learning and to provide enhanced nonlinear separability. Simulation results for well known problems in classification and time-series prediction indicate that the use of ensemble encoding can significantly reduce the time required to train MLP networks. Since the choice of representation for input data is independent of the learning algorithm and the functional form employed in the MLP model, nonlinear preprocessing of network inputs may be an attractive alternative for many MLP network applications.

Receptive field optimization for ensemble encoding

“…We compared the performance of these four methods, in addition to the previously published symmetrical placement method [27], on three wellknown benchmark data sets, with the maximum number of receptors per attribute varying from 3 to 5. To reduce the impact of random noise, we used leave-one-out cross-validation, and repeated all experiments three times.…”

“…Although the symmetrical placement technique was presented in [27] for the case of only three receptors per attribute, it can be generalized to r receptors. For a given attribute, a 1 and a r are set to the minimum and maximum values, respectively, for that attribute; each a i , for i=2,...,rÀ1, is assigned according to …”

“…For instance, Hancock [13] and Lorquet et al [19] have studied preprocessing schemes in which overlapping, Gaussian response curves were used to encode the inputs to MLP networks. Narayan et al [27] proposed a data representation scheme termed ensemble encoding that uses multiple, overlapping receptive fields to generate a distributed representation for network inputs. In the context of some well-known problems in classification and prediction, ensemble encoding was shown [26] to accelerate learning in MLP networks, and promote local learning.…”

“…Previous work by Narayan et al [27] used three receptors for all attributes and used symmetrical placement to assign cluster centers. For each attribute, a 1 and a 3 were assigned to the minimum and the maximum, respectively, data values for that attribute.…”

Enhancing incremental learning in MLP networks using ensemble encoding of network inputs

“…Thus, a network can adapt to one pattern while changing its response to unlike patterns only slightly. This property can lead to local leaning and previous work [7,8] has shown that ensemble encoding of network inputs can accelerate learning in MLP networks. The intent of the current work is to examine the impact of ensemble encoding on the incremental learning ability of MLP networks.…”