Efficiency Comparison of Unstable Transductive and Inductive Conformal Classifiers

Linusson, Henrik; Johansson, Ulf; Boström, Henrik; Löfström, Tuve

doi:10.1007/978-3-662-44722-2_28

Cited by 11 publications

(8 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Conformal prediction has attracted increasing attention in recent years and several studies have investigated both theoretical aspects and applications. [19][20][21][22][23][24][25][26][27][28][29][30][31][32] These studies have highlighted several strengths associated with conformal prediction, such as its excellent handling of imbalanced data, 26,27,33 and built-in definition of the applicability domain. 20,34 Previous studies have also applied conformal prediction to QSAR modelling.…”

Section: Introductionmentioning

confidence: 99%

Conformal Regression for Quantitative Structure–Activity Relationship Modeling—Quantifying Prediction Uncertainty

Svensson

Aniceto

Norinder

et al. 2018

J. Chem. Inf. Model.

View full text Add to dashboard Cite

Making predictions with an associated confidence is highly desirable as it facilitates decision making and resource prioritization. Conformal regression is a machine learning framework that allows the user to define the required confidence and delivers predictions that are guaranteed to be correct to the selected extent. In this study, we apply conformal regression to model molecular properties and bioactivity values and investigate different ways to scale the resultant prediction intervals to create as efficient (i.e., narrow) regressors as possible. Different algorithms to estimate the prediction uncertainty were used to normalize the prediction ranges, and the different approaches were evaluated on 29 publicly available data sets. Our results show that the most efficient conformal regressors are obtained when using the natural exponential of the ensemble standard deviation from the underlying random forest to scale the prediction intervals, but other approaches were almost as efficient. This approach afforded an average prediction range of 1.65 pIC50 units at the 80% confidence level when applied to bioactivity modeling. The choice of nonconformity function has a pronounced impact on the average prediction range with a difference of close to one log unit in bioactivity between the tightest and widest prediction range. Overall, conformal regression is a robust approach to generate bioactivity predictions with associated confidence.

show abstract

Section: Introductionmentioning

confidence: 99%

Conformal Regression for Quantitative Structure–Activity Relationship Modeling—Quantifying Prediction Uncertainty

Svensson

Aniceto

Norinder

et al. 2018

J. Chem. Inf. Model.

View full text Add to dashboard Cite

show abstract

“…• SCP: we fix a calibration split of 20%, as advised in (Linusson et al, 2014). We obtain the nonconformity scores by fitting the model in the proper training set and computing the loss for the calibration data.…”

Section: C3 Design Choices For the Methodsmentioning

confidence: 99%

Approximating Full Conformal Prediction at Scale via Influence Functions

Abad¹,

Bhatt²,

Weller³

et al. 2022

Preprint

View full text Add to dashboard Cite

Conformal prediction (CP) is a wrapper around traditional machine learning models, giving coverage guarantees under the sole assumption of exchangeability; in classification problems, for a chosen significance level ε, CP guarantees that the number of errors is at most ε, irrespective of whether the underlying model is misspecified. However, the prohibitive computational costs of "full" CP led researchers to design scalable alternatives, which alas do not attain the same guarantees or statistical power of full CP. In this paper, we use influence functions to efficiently approximate full CP. We prove that our method is a consistent approximation of full CP, and empirically show that the approximation error becomes smaller as the training set increases; e.g., for 10 3 training points the two methods output p-values that are < 10 −3 apart: a negligible error for any practical application. Our methods enable scaling full CP to large real-world datasets. We compare our full CP approximation (ACP) to mainstream CP alternatives, and observe that our method is computationally competitive whilst enjoying the statistical predictive power of full CP.

show abstract

“…In the experiments, a 10x10-fold cross-validation was performed, and the results presented are averaged across the 10 iterations. In each fold, 25% of the training data was used as the calibration set for the inductive conformal classifier, as suggested in [5].…”

Section: Methodsmentioning

confidence: 99%

Classification with Reject Option Using Conformal Prediction

Linusson

Johansson

Boström

et al. 2018

Advances in Knowledge Discovery and Data Mining

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a practically useful means of interpreting the predictions produced by a conformal classifier. The proposed interpretation leads to a classifier with a reject option, that allows the user to limit the number of erroneous predictions made on the test set, without any need to reveal the true labels of the test objects. The method described in this paper works by estimating the cumulative error count on a set of predictions provided by a conformal classifier, ordered by their confidence. Given a test set and a user-specified parameter k, the proposed classification procedure outputs the largest possible amount of predictions containing on average at most k errors, while refusing to make predictions for test objects where it is too uncertain. We conduct an empirical evaluation using benchmark datasets, and show that we are able to provide accurate estimates for the error rate on the test set.

show abstract

Efficiency Comparison of Unstable Transductive and Inductive Conformal Classifiers

Cited by 11 publications

References 8 publications

Conformal Regression for Quantitative Structure–Activity Relationship Modeling—Quantifying Prediction Uncertainty

Conformal Regression for Quantitative Structure–Activity Relationship Modeling—Quantifying Prediction Uncertainty

Approximating Full Conformal Prediction at Scale via Influence Functions

Classification with Reject Option Using Conformal Prediction

Contact Info

Product

Resources

About