Conformal Prediction with Neural Networks

Abstract

Conformal Prediction (CP) is a method that can be used for complementing the bare predictions produced by any traditional machine learning algorithm with measures of confidence. CP gives good accuracy and confidence values, but unfortunately it is quite computationally inefficient. This computational inefficiency problem becomes huge when CP is coupled with a method that requires long training times, such as Neural Networks. In this paper we use a modification of the original CP method, called Inductive Conformal Prediction (ICP), which allows us to construct a Neural Network confidence predictor without the massive computational overhead of CP. The method we propose accompanies its predictions with confidence measures that are useful in practice, while still preserving the computational efficiency of its underlying Neural Network.

Inductive Conformal Prediction: Theory and Application to Neural Networks

“…The experimental results detailed in Section 5 and in (Papadopoulos et al, 2002a;Papadopoulos et al, 2002b;Papadopoulos et al, 2007) show that the accuracy of ICPs is comparable to that of traditional methods, while the confidence measures they produce are useful in practice. Of course, as a result of removing some examples from the training set to form the calibration set, they sometimes suffer a small, but usually negligible, loss of accuracy from their underlying algorithm.…”

“…Furthermore, the TCP uses a richer set of nonconformity scores, computed from all the training examples, when calculating the p-values for each possible classification, as opposed to the small part of training examples, the calibration set, the ICP uses for the same purpose. As the experimental results in Section 5 and in (Papadopoulos et al, 2002a;Papadopoulos et al, 2002b;Papadopoulos et al, 2007) show, this loss of accuracy is negligible while the improvement in computational efficiency is massive.…”

“…Note that in order to differentiate clearly between the original CP and ICP approaches, the former will be mostly called Transductive Conformal Prediction (TCP). Section 4 explains the application of ICP to Neural Networks, first presented in (Papadopoulos et al, 2007), and Section 5 lists some experimental results obtained by testing the Neural Networks ICP on benchmark data sets. Finally, Section 6 presents the conclusions of the chapter.…”

Conformal predictions in multimedia pattern recognition

“…The limitations of these theories in obtaining practical reliable values of confidence are detailed in [41], [48], [49], [50] and [1], and are summarized below.…”

Efficiency Comparison of Unstable Transductive and Inductive Conformal Classifiers

“…Moreover, as noted in several papers, ICP models typically suffer a small loss in predictive efficiency compared to corresponding TCP models due to the reduced number of training and calibration examples [1,[5][6][7][8]. However, as pointed out in [1], an unstable nonconformity function -one that is heavily influenced by an outlier example, i.e., an erroneously labled test instance (x k+1 ,ỹ) -can cause TCP models to become inefficient.…”