How Good a Shallow Neural Network Is for Solving Non-linear Decision Making Problems

Abstract

The universe approximate theorem states that a shadow neural network (one hidden layer) can represent any non-linear function. In this paper, we aim at examining how good a shadow neural network is for solving non-linear decision making problems. We proposed a performance driven incremental approach to searching the best shadow neural network for decision making, given a data set. The experimental results on the two benchmark data sets, Breast Cancer in Wisconsin and SMS Spams, demonstrate the correction of universe approximate theorem, and show that the number of hidden neurons, taking about the half of input number, is good enough to represent the function from data. It is shown that the performance driven BP learning is faster than the error-driven BP learning, and that the performance of the SNN obtained by the former is not worse than that of the SNN obtained by the latter. This indicates that when learning a neural network with the BP algorithm, the performance reaches a certain value quickly, but the error may still keep reducing. The performance of the SNNs for the two databases is comparable to or better than that of the optimal linguistic attribute hierarchy, obtained by a genetic algorithm in wrapper or in terms of semantics manually, which is much time-consuming.

Learnability and Robustness of Shallow Neural Networks Learned With a Performance-Driven BP and a Variant PSO For Edge Decision-Making

“…As the experiments in [3], the number of hidden neurons of the SNN is increased from 1 to n, where n is the input number of the problem space.…”

“…Hence, a performance-driven backpropagation (PDBP) algorithm and a variant of particle swarm optimization (VPSO) are developed to learn or optimize the weights of an SNN for nonlinear decision making, respectively. Our previous research [3] observed the impact of hidden neuron numbers on the performance of SNNs for different data sets, using an incremental approach to adding a hidden neuron into an SNN gradually. It was shown that when the number of hidden neurons in an SNN for general data sets reaches about half or more of the input number in the problem space, the performance is not improved further or has a tiny change.…”