Assessment of Machine Learning Reliability Methods for Quantifying the Applicability Domain of QSAR Regression Models

Toplak, Marko; Močnik, Rok; Polajnar, Matija; Bosnić, Zoran; Carlsson, Lars; Hasselgren, Catrin; Demšar, Janez; Boyer, Scott; Zupan, Blaž; Stålring, Jonna

doi:10.1021/ci4006595

Cited by 59 publications

(59 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…With this purpose, Das et al 51 introduced the concept of applicability domain in the prediction of binding affinity in docking, although it is commonly used in other research areas such as QSAR. [99][100][101] Zilian and Sotriffer 63 also noted that larger overlaps between training and test sets explained the target-dependent performance of SFCscore RF . Being able to reliably predict how good binding affinity predictions are a priori would be a major advance in docking.…”

Section: Conclusion and Future Prospectsmentioning

confidence: 96%

“…Some authors have explained the performance of a SF a posteriori . With this purpose, Das et al introduced the concept of applicability domain in the prediction of binding affinity in docking, although it is commonly used in other research areas such as QSAR . Zilian and Sotriffer also noted that larger overlaps between training and test sets explained the target‐dependent performance of SFCscore RF .…”

Section: Conclusion and Future Prospectsmentioning

confidence: 99%

See 1 more Smart Citation

Machine‐learning scoring functions to improve structure‐based binding affinity prediction and virtual screening

Ain

Aleksandrova

Roessler

et al. 2015

WIREs Comput Mol Sci

291

234

View full text Add to dashboard Cite

Docking tools to predict whether and how a small molecule binds to a target can be applied if a structural model of such target is available. The reliability of docking depends, however, on the accuracy of the adopted scoring function (SF). Despite intense research over the years, improving the accuracy of SFs for structure‐based binding affinity prediction or virtual screening has proven to be a challenging task for any class of method. New SFs based on modern machine‐learning regression models, which do not impose a predetermined functional form and thus are able to exploit effectively much larger amounts of experimental data, have recently been introduced. These machine‐learning SFs have been shown to outperform a wide range of classical SFs at both binding affinity prediction and virtual screening. The emerging picture from these studies is that the classical approach of using linear regression with a small number of expert‐selected structural features can be strongly improved by a machine‐learning approach based on nonlinear regression allied with comprehensive data‐driven feature selection. Furthermore, the performance of classical SFs does not grow with larger training datasets and hence this performance gap is expected to widen as more training data becomes available in the future. Other topics covered in this review include predicting the reliability of a SF on a particular target class, generating synthetic data to improve predictive performance and modeling guidelines for SF development. WIREs Comput Mol Sci 2015, 5:405–424. doi: 10.1002/wcms.1225For further resources related to this article, please visit the WIREs website.

show abstract

Section: Conclusion and Future Prospectsmentioning

confidence: 96%

Section: Conclusion and Future Prospectsmentioning

confidence: 99%

Machine‐learning scoring functions to improve structure‐based binding affinity prediction and virtual screening

Ain

Aleksandrova

Roessler

et al. 2015

WIREs Comput Mol Sci

291

234

View full text Add to dashboard Cite

show abstract

“…Similarly, a model will not accurately predict activity values outside the range covered in the training data (e.g., predict that a compound is active using a model trained on only inactive compounds). Hence the interest in developing reliable techniques to estimate errors in prediction for individual instances 38,45,46 .…”

Section: Introductionmentioning

confidence: 99%

Discovering Highly Potent Molecules from an Initial Set of Inactives Using Iterative Screening

Cortés-Ciriano

Firth

Bender

et al. 2018

J. Chem. Inf. Model.

View full text Add to dashboard Cite

The versatility of similarity searching and quantitative structure-activity relationships to model the activity of compound sets within given bioactivity ranges (i.e., interpolation) is well established. However, their relative performance in the common scenario in early stage drug discovery where lots of inactive data but no active data points are available (i.e., extrapolation from the low-activity to the high-activity range) has not been thoroughly examined yet. To this aim, we have designed an iterative virtual screening strategy which was evaluated on 25 diverse bioactivity data sets from ChEMBL. We benchmark the efficiency of random forest (RF), multiple linear regression, ridge regression, similarity searching, and random selection of compounds to identify a highly active molecule in the test set among a large number of low-potency compounds. We use the number of iterations required to find this active molecule to evaluate the performance of each experimental setup. We show that linear and ridge regression often outperform RF and similarity searching, reducing the number of iterations to find an active compound by a factor of 2 or more. Even simple regression methods seem better able to extrapolate to high-bioactivity ranges than RF, which only provides output values in the range covered by the training set. In addition, examination of the scaffold diversity in the data sets used shows that in some cases similarity searching and RF require two times as many iterations as random selection depending on the chemical space covered in the initial training data. Lastly, we show using bioactivity data for COX-1 and COX-2 that our framework can be extended to multitarget drug discovery, where compounds are selected by concomitantly considering their activity against multiple targets. Overall, this study provides an approach for iterative screening where only inactive data are present in early stages of drug discovery in order to discover highly potent compounds and the best experimental set up in which to do so.

show abstract

“…Recently, another type of AD measurement was proposed by building an additional error model to assess the prediction reliability [9][10] . Benchmark studies 2,[11][12] on various AD metrics have previously been performed. Toplak et al 12 showed that methods of reliability indices were sensitive to dataset characteristics and to the regression method used in building the QSAR model.…”

Section: Introductionmentioning

confidence: 99%

“…Benchmark studies 2,[11][12] on various AD metrics have previously been performed. Toplak et al 12 showed that methods of reliability indices were sensitive to dataset characteristics and to the regression method used in building the QSAR model. Most of these AD metrics lack a rigorous scientific derivation.…”

Section: Introductionmentioning

confidence: 99%

Applying Mondrian Cross-Conformal Prediction to Estimate Prediction Confidence on Large Imbalanced Bioactivity Datasets

Sun

Carlsson

Ahlberg

et al. 2017

Preprint

Self Cite

View full text Add to dashboard Cite

Conformal prediction has been proposed as a more rigorous way to define prediction confidence compared to other application domain concepts that have earlier been used for QSAR modelling. One main advantage of such a method is that it provides a prediction region potentially with multiple predicted labels, which contrasts to the single valued (regression) or single label (classification) output predictions by standard QSAR . CC-BY-NC-ND 4.0 International license peer-reviewed) is the author/funder. It is made available under aThe copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/116764 doi: bioRxiv preprint first posted online Mar. 15, 2017; 2 modelling algorithms. Standard conformal prediction might not be suitable for imbalanced datasets. Therefore, Mondrian cross-conformal prediction (MCCP) which combines the Mondrian inductive conformal prediction with cross-fold calibration sets has been introduced. In this study, the MCCP method was applied to 18 publicly available datasets that have various imbalance levels varying from 1:10 to 1:1000 (ratio of active/inactive compounds). Our results show that MCCP in general performed well on cheminformatics datasets with various imbalance levels. More importantly, the method not only provides confidence of prediction and prediction regions compared to standard machine learning methods, but also produces valid predictions for the minority class. In addition, a compound similarity based nonconformity measure was investigated. Our results demonstrate that although it gives valid predictions, its efficiency is much worse than nonconformity measures obtained from supervised learning.

show abstract

Assessment of Machine Learning Reliability Methods for Quantifying the Applicability Domain of QSAR Regression Models

Cited by 59 publications

References 22 publications

Machine‐learning scoring functions to improve structure‐based binding affinity prediction and virtual screening

Machine‐learning scoring functions to improve structure‐based binding affinity prediction and virtual screening

Discovering Highly Potent Molecules from an Initial Set of Inactives Using Iterative Screening

Applying Mondrian Cross-Conformal Prediction to Estimate Prediction Confidence on Large Imbalanced Bioactivity Datasets

Contact Info

Product

Resources

About