Neural Network Classifiers Estimate Bayesian a posteriori Probabilities

Abstract

Many neural network classifiers provide outputs which estimate Bayesian a posteriori probabilities. When the estimation is accurate, network outputs can be treated as probabilities and sum to one. Simple proofs show that Bayesian probabilities are estimated when desired network outputs are 1 of M (one output unity, all others zero) and a squared-error or cross-entropy cost function is used. Results of Monte Carlo simulations performed using multilayer perceptron (MLP) networks trained with backpropagation, radial basis function (RBF) networks, and high-order polynomial networks graphically demonstrate that network outputs provide good estimates of Bayesian probabilities. Estimation accuracy depends on network complexity, the amount of training data, and the degree to which training data reflect true likelihood distributions and a priori class probabilities. Interpretation of network outputs as Bayesian probabilities allows outputs from multiple networks to be combined for higher level decision making, simplifies creation of rejection thresholds, makes it possible to compensate for differences between pattern class probabilities in training and test data, allows outputs to be used to minimize alternative risk functions, and suggests alternative measures of network performance.

Neural network wind retrieval from ERS‐1 scatterometer data

“…2. For A-NNi, the simple quadratic function implies we obtain an approximation of posterior probabilities [Richard and Lippman, 1991]. This cost function choice has also proven to be efficient in the ambiguity removal phase.…”

Smart Feature Detection using an Invariance Network Architecture.

“…Under these conditions a network trained with the least squares error function will approximate the conditional probability of classification. For an extended proof of this theorem the reader is directed to [14] …”

Characterizing major agricultural land change trends in the Western Corn Belt

“…Many classification algorithms can be trained to generate fuzzy outputs or approximate posterior probabilities (Richard and Lippmann, 1991;Moody et al, 1996;). An important practice within the machine learning community is that classification performance can be significantly improved by simply adjusting the output threshold value instead of using a default one (e.g., 0.5) to label each target class (Provost, 2000;Zhou and Liu, 2006).…”