Optimal Learning with a Neural Network

Abstract

We introduce optimal learning with a neural network, which we define as minimising the expectation generalisation error. We find that the optimally-trained spherical perceptron may learn a linearly-separable rule as well as any possible network. We sketch an algorithm to generate optimal learning, and simulation results support our conclusions. Optimal learning of a well-known, significant unlearnable problem, the <<mismatched weight. problem, gives better asymptotic learning than conventional techniques, and may be simulated enormously more easily. Unlike many other learning schemes, optimal learning extends to more general networks learning more complex rules.

Finite size scaling of the Bayesian perceptron

“…With this cost function, the optimal generalizer may be found by a simple gradient descent, with neither the need to train an infinite number of perceptrons for implementing a commitee machine, as was suggested by Opper and Haussler [8], nor to determine a large number of 'samplers' of the version space, as proposed by Watkin [10]. Once the potential is known, it is straightforward to calculate the distribution of stabilities of the training set:…”

“…The fact that the bayesian student has patterns at vanishing distance from the hyperplane, and has most patterns at distances larger than κ, allows us to conclude that its weight vector lies close to the boundary of the version space. It has been shown [10] that the bayesian weight vector is the barycenter of the (strictly convex) version space. Our result means that the barycenter of the version space is far from its center, which is rather surprising, and might indicate that the version space is highly non-spherical.…”

“…The generalization performance of these algorithms is very close to the bayesian optimal value. Some of them end up with a finite fraction of training errors, suggesting that the optimal solution might lie outside the version space, but it has been established that this is not the case [10]. More recently, the cost function that minimizes the generalization error, was determined through a variational approach, and it was showed that its minimum endows the perceptron with the optimal, bayesian, generalization performance [11].…”

Calibrating artificial neural networks by global optimization

“…ANN application areas includes data classification and pattern recognition (Ripley, 1996), damage detection and earthquake simulation (Pei et al, 2006), function approximation (Toh, 1999;Ye and Lin, 2003), material science (Bhadeshia, 1999), experimental design of engineering systems (Röpke et al, 2005), nonlinear optimization (Malek et al, 2010), polypeptide structure prediction (Dorn and de Souza, 2010), prediction of trading signals of stock market indices (Tilakaratne et al, 2008), regression analysis (De Veux et al, 1998), signal and image processing (Watkin, 1993;Masters, 1994), time series analysis and forecasting (Franses and van Dijk, 2000;Kajitani et al, 2005).…”

“…ANN model parameterization frameworks and numerical studies are presented and discussed e.g. by Watkin (1993), Prechelt (1994), Bianchini and Gori (1996), Sexton et al (1998), Jordanov and Brown (1999), Toh (1999), Ye and Lin (2003), Abraham (2004), Hamm et al (2007).…”