Conformal Regression for Quantitative Structure–Activity Relationship Modeling—Quantifying Prediction Uncertainty

Svensson, Fredrik; Aniceto, Natália; Norinder, Ulf; Cortés-Ciriano, Isidro; Spjuth, Ola; Carlsson, Lars; Bender, Andreas

doi:10.1021/acs.jcim.8b00054

Cited by 51 publications

(82 citation statements)

References 54 publications

(103 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To improve reliability of the models, the chemical space was considered by including information about the nearest neighbours to normalise the conformal predictions. While such a normalisation of the nc scores is important for regression models [56,57], to the best of our knowledge, it has not been applied to classification tasks so far. Including the KNN normalisation clearly improved validity for internal validation and the in-house dataset from 0.81 to 0.85 and from 0.59 to 0.82 at 0.2 SL, respectively (see Additional file 1: Table S2).…”

Section: Conformal Predictors-validation Of Aa Modelmentioning

confidence: 99%

KnowTox: pipeline and case study for confident prediction of potential toxic effects of compounds in early phases of development

Morger¹,

Mathea²,

Achenbach³

et al. 2020

J Cheminform

View full text Add to dashboard Cite

Risk assessment of newly synthesised chemicals is a prerequisite for regulatory approval. In this context, in silico methods have great potential to reduce time, cost, and ultimately animal testing as they make use of the ever-growing amount of available toxicity data. Here, KnowTox is presented, a novel pipeline that combines three different in silico toxicology approaches to allow for confident prediction of potentially toxic effects of query compounds, i.e. machine learning models for 88 endpoints, alerts for 919 toxic substructures, and computational support for read-across. It is mainly based on the ToxCast dataset, containing after preprocessing a sparse matrix of 7912 compounds tested against 985 endpoints. When applying machine learning models, applicability and reliability of predictions for new chemicals are of utmost importance. Therefore, first, the conformal prediction technique was deployed, comprising an additional calibration step and per definition creating internally valid predictors at a given significance level. Second, to further improve validity and information efficiency, two adaptations are suggested, exemplified at the androgen receptor antagonism endpoint. An absolute increase in validity of 23% on the in-house dataset of 534 compounds could be achieved by introducing KNNRegressor normalisation. This increase in validity comes at the cost of efficiency, which could again be improved by 20% for the initial ToxCast model by balancing the dataset during model training. Finally, the value of the developed pipeline for risk assessment is discussed using two in-house triazole molecules. Compared to a single toxicity prediction method, complementing the outputs of different approaches can have a higher impact on guiding toxicity testing and de-selecting most likely harmful development-candidate compounds early in the development process.

show abstract

Section: Conformal Predictors-validation Of Aa Modelmentioning

confidence: 99%

KnowTox: pipeline and case study for confident prediction of potential toxic effects of compounds in early phases of development

Morger¹,

Mathea²,

Achenbach³

et al. 2020

J Cheminform

View full text Add to dashboard Cite

show abstract

“…Each QSAR model was validated using both internal (i.e., cross-validated) and external (i.e., test set) error measures and only models that satisfied stringent quality criteria were used for the construction of the rv-QAFFP fingerprint. The applicability domain of individual QSAR models was estimated using inductive conformal prediction [54][55][56][57]. The rv-QAFFP fingerprint is composed of 440 affinities predicted for the panel of assays covering 376 distinct molecular targets models, further referred to as point prediction models, out of the initial set of 1360 models were considered to be reliable and were used for the construction of the rv-QAFFP fingerprint (Additional file 1).…”

Section: Rv-qaffp Fingerprint Constructionmentioning

confidence: 99%

QSAR-derived affinity fingerprints (part 1): fingerprint construction and modeling performance for similarity searching, bioactivity classification and scaffold hopping

et al. 2020

Self Cite

View full text Add to dashboard Cite

An affinity fingerprint is the vector consisting of compound's affinity or potency against the reference panel of protein targets. Here, we present the QAFFP fingerprint, 440 elements long in silico QSAR-based affinity fingerprint, components of which are predicted by Random Forest regression models trained on bioactivity data from the ChEMBL database. Both real-valued (rv-QAFFP) and binary (b-QAFFP) versions of the QAFFP fingerprint were implemented and their performance in similarity searching, biological activity classification and scaffold hopping was assessed and compared to that of the 1024 bits long Morgan2 fingerprint (the RDKit implementation of the ECFP4 fingerprint). In both similarity searching and biological activity classification, the QAFFP fingerprint yields retrieval rates, measured by AUC (~ 0.65 and ~ 0.70 for similarity searching depending on data sets, and ~ 0.85 for classification) and EF5 (~ 4.67 and ~ 5.82 for similarity searching depending on data sets, and ~ 2.10 for classification), comparable to that of the Mor-gan2 fingerprint (similarity searching AUC of ~ 0.57 and ~ 0.66, and EF5 of ~ 4.09 and ~ 6.41, depending on data sets, classification AUC of ~ 0.87, and EF5 of ~ 2.16). However, the QAFFP fingerprint outperforms the Morgan2 fingerprint in scaffold hopping as it is able to retrieve 1146 out of existing 1749 scaffolds, while the Morgan2 fingerprint reveals only 864 scaffolds.

show abstract

“…Throughout this work we considered a Confidence Level of 80% unless otherwise stated, as this confidence level in our experience represents a generally suitable trade-off between efficiency and validity 39 .…”

Section: Conformal Prediction -Dropout Conformal Predictionmentioning

confidence: 99%

“…revealed that the exponential scaling improves the efficiency of Conformal Predictors built on bioactivity data sets39 . This scaling sets the upper value for the list of nonconformity values to be equal to the largest residual in the validation set, as the exponential converts low 0 values to values close to unity.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout

Cortés-Ciriano

Bender

2019

J. Chem. Inf. Model.

Self Cite

View full text Add to dashboard Cite

While the use of deep learning in drug discovery is gaining increasing attention, the lack of methods to computate reliable errors in prediction for Neural Networks prevents their application to guide decision making in domains where identifying unreliable predictions is essential, e.g. precision medicine. Here, we present a framework to compute reliable errors in prediction for Neural Networks using Test-Time Dropout and Conformal Prediction. Specifically, the algorithm consists of training a single Neural Network using dropout, and then applying it N times to both the validation and test sets, also employing dropout in this step. Therefore, for each instance in the validation and test sets an ensemble of predictions were generated. The residuals and absolute errors in prediction for the validation set were then used to compute prediction errors for test set instances using Conformal Prediction. We show using 24 bioactivity data sets from ChEMBL 23 that dropout Conformal Predictors are valid (i.e., the fraction of instances whose true value lies within the predicted interval strongly correlates with the confidence level) and efficient, as the predicted confidence intervals span a narrower set of values than those computed with Conformal Predictors generated using Random Forest (RF) models. Lastly, we show in retrospective virtual screening experiments that dropout and RF-based Conformal Predictors lead to comparable retrieval rates of active compounds. Overall, we propose a computationally efficient framework (as only N extra forward passes are required in addition to training a single network) to harness Test-Time Dropout and the Conformal Prediction framework, and to thereby generate reliable prediction errors for deep Neural Networks. Machine Learning -Data SplittingThe data sets were randomly split into a training set (70% of the data), a validation set (15%), and a test set (15%). For each data set, the training set was used to train a given network, whereas the validation set served to monitor the performance of the network during the training phase. In case of RF models, both the training and validation sets were used for model training.The predictive power of the final RF and DNN model was evaluated on the test set. The above split (and associated model training and testing) was repeated 20 times with random data set assignments. -Deep Neural Networks (DNN)DNNs were trained using the python library Pytorch 48 . We defined four hidden layers, composed of 1000, 1000, 100 and 10 nodes, respectively. The number of neurons in each layer was selected to be smaller than the input fingerprint size to reduce the chances of overfitting 49 .Rectified linear unit (ReLU) activation was used in all cases. The training data was processed in batches of size equal to 15% of the number of instances. We used Stochastic Gradient Descent with Nesterov momentum, which was set to 0.9 and kept constant during the training phase 50 .The networks were trained over 4,000 epochs, and early stopping was used in all cases, i.e., the ...

show abstract

Conformal Regression for Quantitative Structure–Activity Relationship Modeling—Quantifying Prediction Uncertainty

Cited by 51 publications

References 54 publications

KnowTox: pipeline and case study for confident prediction of potential toxic effects of compounds in early phases of development

KnowTox: pipeline and case study for confident prediction of potential toxic effects of compounds in early phases of development

QSAR-derived affinity fingerprints (part 1): fingerprint construction and modeling performance for similarity searching, bioactivity classification and scaffold hopping

Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout

Contact Info

Product

Resources

About