Predicting With Confidence: Using Conformal Prediction in Drug Discovery

Alvarsson, Jonathan; Arvidsson, Staffan; Norinder, Ulf; Spjuth, Ola

doi:10.1016/j.xphs.2020.09.055

Cited by 72 publications

(64 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The prediction set contains all class labels, for which the p-value is larger than the significance level. For more information on conformal prediction, see Alvarsson et al [40] and Norinder et al [41].…”

Section: Conformal Predictionmentioning

confidence: 99%

Assessing the calibration in toxicological in vitro models with conformal prediction

et al. 2021

Self Cite

View full text Add to dashboard Cite

Machine learning methods are widely used in drug discovery and toxicity prediction. While showing overall good performance in cross-validation studies, their predictive power (often) drops in cases where the query samples have drifted from the training data’s descriptor space. Thus, the assumption for applying machine learning algorithms, that training and test data stem from the same distribution, might not always be fulfilled. In this work, conformal prediction is used to assess the calibration of the models. Deviations from the expected error may indicate that training and test data originate from different distributions. Exemplified on the Tox21 datasets, composed of chronologically released Tox21Train, Tox21Test and Tox21Score subsets, we observed that while internally valid models could be trained using cross-validation on Tox21Train, predictions on the external Tox21Score data resulted in higher error rates than expected. To improve the prediction on the external sets, a strategy exchanging the calibration set with more recent data, such as Tox21Test, has successfully been introduced. We conclude that conformal prediction can be used to diagnose data drifts and other issues related to model calibration. The proposed improvement strategy—exchanging the calibration data only—is convenient as it does not require retraining of the underlying model.

show abstract

Section: Conformal Predictionmentioning

confidence: 99%

Assessing the calibration in toxicological in vitro models with conformal prediction

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…CP models, or generally ML models, are widely used for molecular property predictions, including activity or toxicity 5,6,55 . Notably, the CP framework is based on the assumption that test and calibration data stem from the same distribution 10,11 . If this prerequisite is not given, the models are not guaranteed to be valid (i.e.…”

Section: Discussionmentioning

confidence: 99%

Studying and Mitigating the Effects of Data Drifts on ML Model Performance at the Example of Chemical Toxicity Data

Morger¹,

Lomana²,

Norinder³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning models are widely applied to predict molecular properties or the biological activity of small molecules on a specific protein. Models can be integrated in a conformal prediction (CP) framework which adds a calibration step to estimate the confidence of the predictions. CP models present the advantage of ensuring a predefined error rate under the assumption that test and calibration set are exchangeable. In cases where the test data have drifted away from the descriptor space of the training data, or where assay setups have changed, this assumption might not be fulfilled and the models are not guaranteed to be valid. In this study, the performance of internally valid CP models when applied to either newer time-split data or to external data was evaluated. In detail, temporal data drifts were analysed based on twelve datasets from the ChEMBL database. In addition, discrepancies between models trained on publicly available data and applied to proprietary data for the liver toxicity and MNT in vivo endpoints were investigated. In most cases, a drastic decrease in the validity of the models was observed when applied to the time-split or external (holdout) test sets. To overcome the decrease in model validity, a strategy for updating the calibration set with data more similar to the holdout set was investigated. Updating the calibration set generally improved the validity, restoring it completely to its expected value in many cases. The restored validity is the first requisite for applying the CP models with confidence. However, the increased validity comes at the cost of a decrease in model efficiency, as more predictions are identified as inconclusive. This study presents a strategy to recalibrate CP models to mitigate the effects of data drifts. Updating the calibration sets without having to retrain the model has proven to be a useful approach to restore the validity of most models.

show abstract

“…The prediction set contains all class labels, for which the p-value is larger than the significance level. For more information on conformal prediction, see Alvarsson et al [32] and Norinder et al [33].…”

Section: Modelling 221 Conformal Predictionmentioning

confidence: 99%

Assessing the Calibration in Toxicological in Vitro Models with Conformal Prediction

Morger

Svensson

Arvidsson³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning methods are widely used in drug discovery and toxicity prediction. While showing overall good performance in cross-validation studies, their predictive power (often) drops in cases where the query samples have drifted from the training data’s descriptor space. Thus, the assumption for applying machine learning algorithms, that training and test data stem from the same distribution, might not always be fulfilled. In this work, conformal prediction is used to assess the calibration of the models. Deviations from the expected error may indicate that training and test data originate from different distributions. Exemplified on the Tox21 datasets, composed of chronologically released Tox21Train, Tox21Test and Tox21Score subsets, we observed that while internally valid models could be trained using cross-validation on Tox21Train, predictions on the external Tox21Score data resulted in higher error rates than expected. To improve the prediction on the external sets, a strategy exchanging the calibration set with more recent data, such as Tox21Test, has successfully been introduced. We conclude that conformal prediction can be used to diagnose data drifts and other issues relating to model calibration. The proposed improvement strategy — exchanging the calibration data only — is convenient as it does not require retraining of the underlying model.

show abstract

Predicting With Confidence: Using Conformal Prediction in Drug Discovery

Cited by 72 publications

References 22 publications

Assessing the calibration in toxicological in vitro models with conformal prediction

Assessing the calibration in toxicological in vitro models with conformal prediction

Studying and Mitigating the Effects of Data Drifts on ML Model Performance at the Example of Chemical Toxicity Data

Assessing the Calibration in Toxicological in Vitro Models with Conformal Prediction

Contact Info

Product

Resources

About