Traditional Machine and Deep Learning for Predicting Toxicity Endpoints

Abstract

Molecular structure property modeling is an increasingly important tool for predicting compounds with desired properties due to the expensive and resource-intensive nature and the problem of toxicity-related attrition in late phases during drug discovery and development. Lately, the interest for applying deep learning techniques has increased considerably. This investigation compares the traditional physico-chemical descriptor and machine learning-based approaches through autoencoder generated descriptors to two different descriptor-free, Simplified Molecular Input Line Entry System (SMILES) based, deep learning architectures of Bidirectional Encoder Representations from Transformers (BERT) type using the Mondrian aggregated conformal prediction method as overarching framework. The results show for the binary CATMoS non-toxic and very-toxic datasets that for the former, almost equally balanced, dataset all methods perform equally well while for the latter dataset, with an 11-fold difference between the two classes, the MolBERT model based on a large pre-trained network performs somewhat better compared to the rest with high efficiency for both classes (0.93–0.94) as well as high values for sensitivity, specificity and balanced accuracy (0.86–0.87). The descriptor-free, SMILES-based, deep learning BERT architectures seem capable of producing well-balanced predictive models with defined applicability domains. This work also demonstrates that the class imbalance problem is gracefully handled through the use of Mondrian conformal prediction without the use of over- and/or under-sampling, weighting of classes or cost-sensitive methods.

CPSign: conformal prediction for cheminformatics modeling

“…As conformal prediction produces prediction intervals, traditional metrics such as AUC and F1 cannot be used, and we instead use the well-established metrics of Observed Fuzziness, Average C and fraction of single-label prediction intervals. Using mondrian conformal prediction [ 6 ] also improves the modeling and calibration for imbalanced datasets, which has been shown in several ligand-based studies [ 18 , 31 , 32 ].…”

“…Under the relatively week assumption of exchangeability of calibration and test-data, these inductive versions of conformal predictors are proven to produce valid (well-calibrated) predictions, i.e., that the error rate is equal to or smaller than the specified significance level [ 6 ]. Furthermore, in the case of classification, given that a mondrian (class conditional) calibration is used, the guarantee holds individually for each class and has been shown to handle imbalanced datasets very well without the need to apply balancing techniques [ 18 , 31 , 32 ]. However, this guarantee may in practice be difficult to achieve sometimes due to assay drifts [ 12 ] or in case data splitting is performed in a non-random way (such as scaffold-splitting).…”

CPSign - Conformal Prediction for Cheminformatics Modeling

“…As conformal prediction produces prediction intervals, traditional metrics such as AUC and Fi cannot be used, and we instead use the well-established metrics of Observed Fuzziness, Average C and fraction of single-label prediction intervals. Using mondrian conformal prediction [6] also improves the modeling and calibration for imbalanced datasets, which has been shown in several ligand-based studies [30,18, 31].…”

“…Under the relatively week assumption of exchangeability of calibration and test-data, these inductive versions of conformal predictors are proven to produce valid (well-calibrated) predictions, i.e., that the error rate is equal to or smaller than the specified significance level [6]. Furthermore, in the case of classification, given that a mondrian (class conditional) calibration is used, the guarantee holds individually for each class and has been shown to handle imbalanced datasets very well without the need to apply balancing techniques [30, 31, 18]. However, this guarantee may in practice be difficult to achieve sometimes due to assay drifts [12] or in case data splitting is performed in a non-random way (such as scaffold splitting).…”

“…Conformal prediction has been extensively used in drug discovery [9] with applications including screening [10], toxicology prediction [11, 12], property prediction [13], target prediction [14], and prediction of pharmacokinetics [15, 16]. More recently, conformal prediction has also been used with Deep Neural Networks in drug discovery applications [17, 18, 19] and in medical applications [20].…”