A reply to Honavar's book review of Neural Network Design and the Complexity of Learning

Abstract

I want to take this opportunity to reinforce a point that is perhaps not stated clearly enough in my book. It is a point that Honavar (and another reviewer) have rightfully pursued.The NP-completeness is with respect to the size of the network and the amount of data in the training set to be loaded. Honavar points out that this measure of complexity of the input is somewhat suspect because "it does not reflect the.., inherent regularities of the data items being loaded; nor can it capture possible effects of choosing a particular network topology for loading a given set of data items." Very true. If we let the devil specify the network architecture (and the data to be loaded) then he can make our lives exceedingly difficult, but we should not take this result to indicate that our lives have to be difficult.In the domain of connectionist research, it is the researcher who gets to play the role of the devil, and he is certainly not trying to be malicious. If he knows what he is doing, he can presumably make the problem easy; but, no one yet has a theory of how best to accomplish this. The themes of my results are that it is easy to get into trouble, and that we need to elaborate the theory further. (Maybe the common all-to-all connectivity scheme is a solution for small nets, but that claim has to be proved and it may introduce concomitant generalization problems.)In the neurological domain, the network design is not under our control; it is nature that has played the role of devil and we must see what hand she has dealt us. For my explorations I picked a particular design family (shallow nets, grid-like structures) to model what seems to be the first-order aspect of her handiwork. Am I off base here? No one has yet suggested so. Even though the architectures are extremely regular, the intractability still shines through, and we are left to conclude that the only types of data that can be loaded in there have to have some kind of fortuitous peculiarity. Honovar's concern about "the interactions between the network architecture and intrinsic structure of the data set to be loaded" are natural and important. My theorems on simple regular architectures are an attempt to address this concern, although such is not well elaborated in the book.Blum and Rivest responded to my early theorems (which had large and complicated architectures as well as specially constructed data sets) by finding an apparently smaller, very simple, and very regular architecture that nevertheless also has intractable data sets. They too were worried that the intractability was somehow due to a devious mismatch between the data and the architecture, but their result also seems to mute this hypothesis. However, neither they nor myself have any crisp statements about this.

Connectionist semantic systematicity

“…Second, the time required for the network to learn the required connection weights may become unrealistically long (Judd, 1990).…”

Training a 3-node neural network is NP-complete

“…The work in this paper is inspired by Judd (1990) who shows the following problem to be NP-complete: "Given a neural network and a set of training examples, does there exist a set of edge weights for the network so that the network produces the correct output for all the training examples?" Judd shows that the problem remains NP-complete even if the network is only required to produce the correct output for two-thirds of the training examples, which implies that even approximately training a neural network is intrinsically difficult in the worst case (Judd, 1988).…”

Computing with Spiking Neuron Networks

“…in [15,78]. Moreover, the results presented in this section apply only to a restricted set of SNN models and, apart from the programmability of transmission delays of synaptic connections, they do not cover all the capabilities of SNNs that could result from computational units based on firing times.…”

“…• Calculability: NNs computational power outperforms a Turing machine [154] • Complexity: The "loading problem" is NP-complete [15,78] • Capacity: MLP, RBF and WNN 1 are universal approximators [35,45,63] • Regularization theory [132]; PAC-learning 2 [171]; Statistical learning theory, VC-dimension, SVM 3 [174] Nevertheless, traditional neural networks suffer from intrinsic limitations, mainly for processing large amount of data or for fast adaptation to a changing environment. Several characteristics, such as iterative learning algorithms or artificially designed neuron model and network architecture, are strongly restrictive compared with biological processing in natural neural networks.…”

“…Then, the computational complexity of training a spiking neuron can be analyzed within the formulation of the consistency or loading problem (cf. [78]):…”