Confidence estimation methods for neural networks: a practical comparison

Abstract

Feedforward neural networks, particularly multilayer perceptrons, are widely used in regression and classification tasks. A reliable and practical measure of prediction confidence is essential. In this work three alternative approaches to prediction confidence estimation are presented and compared. The three methods are the maximum likelihood, approximate Bayesian, and the bootstrap technique. We consider prediction uncertainty owing to both data noise and model parameter misspecification. The methods are tested on a number of controlled artificial problems and a real, industrial regression application, the prediction of paper "curl". Confidence estimation performance is assessed by calculating the mean and standard deviation of the prediction interval coverage probability. We show that treating data noise variance as a function of the inputs is appropriate for the curl prediction task. Moreover, we show that the mean coverage probability can only gauge confidence estimation performance as an average over the input space, i.e., global performance and that the standard deviation of the coverage is unreliable as a measure of local performance. The approximate Bayesian approach is found to perform better in terms of global performance.

Tool wear monitoring and selection of optimum cutting conditions with progressive tool wear effect and input uncertainties

“…The neural network introduces uncertainty due to model misspecification and the inefficiencies of the training method (Papadopoulos and Edwards 2001). A network trained on a given dataset forms a better representation of the data in regions of high input data density (Papadopoulos and Edwards 2001). Moreover, because of the nature of the training algorithm, there is no guarantee that the weight values correspond to the global minimum of the error function.…”

“…Moreover, because of the nature of the training algorithm, there is no guarantee that the weight values correspond to the global minimum of the error function. Even if the global minimum is found, the solution will not necessarily be optimal because the finite training set does not fully describe the true data-generating mechanism (Papadopoulos and Edwards 2001).…”

“…Several approaches have been developed for output confidence interval predications (Papadopoulos and Edwards 2001); they all adopt a probabilistic standpoint and therefore suffer from the common drawback that, since the probability distributions are usually estimated from the low-order moments of the data (typically mean and standard deviation), there is often no validation of the extremes of the distributions. An alternative approach to this issue is the application of an interval-based non-probabilistic (Papadopoulos and Edwards 2001) technique or the MonteCarlo technique to evaluate the robustness of the neural network to uncertainty in input data. The interval-based technique is posited on the theory of information-gap uncertainty and its advantage lies in presenting both crisp data, and interval (Ben-Haim 2001) data to a number of neural networks under evaluation.…”

“…Data noise is not the only source of uncertainty. The neural network introduces uncertainty due to model misspecification and the inefficiencies of the training method (Papadopoulos and Edwards 2001). A network trained on a given dataset forms a better representation of the data in regions of high input data density (Papadopoulos and Edwards 2001).…”

Development of a Committee of Artificial Neural Networks for the Performance Testing of Compressors for Thermal Machines in Very Reduced Times

“…A strategy to minimize these deviations is to use such an ANN committee that the simple average of the ANN outputs is favourable as it gets closer to the expected value [14−17]. The idea behind a committee machine is the minimization of the random effects induced in the ANN due to the learning process [8,18]. It is possible to make a direct analogy with the metrology where, in order to determine the value for a measurand with the presence of random effects in the measurement process, the average of several measurements represents a better result than a single individual measurement [19].…”