Off-targetP ML: an open source machine learning framework for off-target panel safety assessment of small molecules

Abstract

Unpredicted drug safety issues constitute the majority of failures in the pharmaceutical industry according to several studies. Some of these preclinical safety issues could be attributed to the non-selective binding of compounds to targets other than their intended therapeutic target, causing undesired adverse events. Consequently, pharmaceutical companies routinely run in-vitro safety screens to detect off-target activities prior to preclinical and clinical studies. Hereby we present an open source machine learning framework aiming at the prediction of our in-house 50 off-target panel activities for ~ 4000 compounds, directly from their structure. This framework is intended to guide chemists in the drug design process prior to synthesis and to accelerate drug discovery. We also present a set of ML approaches that require minimum programming experience for deployment. The workflow incorporates different ML approaches such as deep learning and automated machine learning. It also accommodates popular issues faced in bioactivity predictions, as data imbalance, inter-target duplicated measurements and duplicated public compound identifiers. Throughout the workflow development, we explore and compare the capability of Neural Networks and AutoML in constructing prediction models for fifty off-targets of different protein classes, different dataset sizes, and high-class imbalance. Outcomes from different methods are compared in terms of efficiency and efficacy. The most important challenges and factors impacting model construction and performance in addition to suggestions on how to overcome such challenges are also discussed.

Identifying Differences in the Performance of Machine Learning Models for Off-Targets Trained on Publicly Available and Proprietary Data Sets

“…Notably, only 4 out of 41 targets (AChE, COX-2, MAO-A, PDE4D) had both a percentage higher than 16 for inactive compounds at a threshold of pChEMBL 5 (see Table ) and a data set size of more than 1000 compounds. In contrast, data sets derived from industry in-house off-target screens show an opposite picture . This poses the question if the high positive data bias present in ChEMBL (when thresholds for off-target screens are used) would bias the model creation toward the overprediction of active compounds, and vice versa for models derived from industry off-target screens.…”

“…At the same time, given coordinates, one can visualize the predictions as a scatter plot using a color scale to represent any additional information layer. By utilizing this approach, one can select an off-target and choose between models based on public data and the models from Naga et al trained on industry data . In Figure a screenshot of the visualization tool is provided.…”

“…Molecular Descriptors. In order to have a similar representation of the molecular descriptors as in the work of Naga et al, 19 extended connectivity fingerprint 4 (ECFP4) 26 was chosen. The 1024 bits of ECFP4 were calculated for each compound from the standardized canonical SMILES using RDKit inside a Python Script KNIME node.…”

“…To the best of our knowledge, this is the first study Data Set Preparation from ChEMBL. Data sets for targets described by Naga et al 19 were extracted from ChEMBL (ChEMBL30, 20 ChEMBL31 21 ), standardized and curated via the UNIVIE ChEMBL Retriever Jupyter Notebook (https://github.com/ PharminfoVienna/cmpOffTargets). Only data points with human, single protein, and IC 50 or K i entries that shared the same gene name as in the work of Naga et al were chosen for further curation and standardization steps.…”

“…The retrieved data from DrugBank were further curated and standardized using the UNIVIE ChEMBL Retriever Jupyter Notebook.Statistical Metrics. To estimate the performance of the binary classification models and to put them into context with those from Naga et al,19 the following parameters were used:•Sensitivity: TP/(TP + FN) • Specificity: TN/(TN + FP) • Balanced accuracy: (sensitivity + specificity)/2 • Accuracy: (TP + TN)/(TP + FP + TN + FN) • Precision or positive predictivity (PPV): TP/(TP + FP) • Negative predictivity (NPV): TN/(TN + FN) • Recall: TP/(TP + FN) • F-Measure: 2 × ((precision × recall)/(precision + recall)) • Matthews correlation coefficient (MCC): (TP × TN − FP × FN)/sqrt((TP + FN)(TP + FP)(TN + FP)(TN + FN)) Modeling Pipeline.…”