Evolved decision trees as conformal predictors

Abstract

Abstract-In conformal prediction, predictive models output sets of predictions with a bound on the error rate. In classification, this translates to that the probability of excluding the correct class is lower than a predefined significance level, in the long run. Since the error rate is guaranteed, the most important criterion for conformal predictors is efficiency. Efficient conformal predictors minimize the number of elements in the output prediction sets, thus producing more informative predictions. This paper presents one of the first comprehensive studies where evolutionary algorithms are used to build conformal predictors. More specifically, decision trees evolved using genetic programming are evaluated as conformal predictors. In the experiments, the evolved trees are compared to decision trees induced using standard machine learning techniques on 33 publicly available benchmark data sets, with regard to predictive performance and efficiency. The results show that the evolved trees are generally more accurate, and the corresponding conformal predictors more efficient, than their induced counterparts. One important result is that the probability estimates of decision trees when used as conformal predictors should be smoothed, here using the Laplace correction. Finally, using the more discriminating Brier score instead of accuracy as the optimization criterion produced the most efficient conformal predictions.

Criteria of Efficiency for Conformal Prediction

“…The S criterion was introduced in [3] and the N criterion was introduced independently in [5] and [3], although the analogue of the N criterion for regression (where the size of a prediction set is defined to be its Lebesgue measure) had been used earlier in [11] (whose arXiv version was published in 2012).…”

Regression conformal prediction with random forests

“…On a number of real-world data sets, however, the predictive regions produced by GP-regression were no longer valid, i.e., they may become misleading when the correct prior is not known. The CP framework has been applied to classification using several popular learning algorithms, such as ANNs (Papadopoulos 2008), kNN (Nguyen and Luo 2012), SVMs (Devetyarov and Nouretdinov 2010;Makili et al 2011), decision trees (Johansson et al 2013a), random forests (Bhattacharyya 2011;Devetyarov and Nouretdinov 2010) and evolutionary algorithms (Johansson et al 2013b;Lambrou et al 2011). Although we in this study consider regression tasks, there is some overlap with previous studies on classification when it comes to design choices.…”

Criteria of efficiency for set-valued classification

“…This paper points out that the standard criteria of efficiency used in literature have a serious disadvantage, and we define a class of criteria of efficiency, called "probabilistic", that do not share this disadvantage (see the discussion at the end of Section 5). In two recent papers [3,5] two probabilistic criteria have been introduced, and in this paper we introduce two more and argue that probabilistic criteria should be used in place of more standard ones. We concentrate on the case of classification only (the label space is finite).…”

“…The other two criteria that had been used before the publication of the conference version [18] of this paper are the sum of the p-values for all potential labels (this does not depend on the significance level) and the size of the prediction set at a given significance level: see the papers [3] and [5].…”