Exposing the limitations of molecular machine learning with activity cliffs.

Abstract

Machine learning has become a crucial tool in drug discovery and chemistry at large, e.g., to predict molecular properties, such as bioactivity, with high accuracy. However, activity cliffs – pairs of molecules that are highly similar in their structure but exhibit large differences in potency – have been underinvestigated for their effect on model performance. Not only are these edge cases informative for molecule discovery and optimization, but models that are well-equipped to accurately predict the potency of activity cliffs have an increased potential for prospective applications. Our work aims to fill the current knowledge gap on best-practice machine learning methods in the presence of activity cliffs. We benchmarked a total of 720 machine and deep learning models on curated bioactivity data from 30 macromolecular targets for their performance on activity cliff compounds. While all methods struggled in the presence of activity cliffs, machine learning approaches based on molecular descriptors outperformed more complex deep learning methods. Our findings highlight large case-by-case differences in performance, advocating for (a) the inclusion of dedicated “activity-cliff-centered” metrics during model development and evaluation, and (b) the development of novel algorithms to better predict the properties of activity cliffs. To this end, the methods, metrics, and results of this study have been encapsulated into an open-access benchmarking platform named MoleculeACE (Activity Cliff Estimation, available on GitHub at: https://github.com/molML/MoleculeACE). MoleculeACE is designed to steer the community towards addressing the pressing but overlooked limitation of molecular machine learning models posed by activity cliffs.

Roughness of Molecular Property Landscapes and Its Impact on Modellability

“…TDC refers to 12 data sets used as regression benchmarks and provided by the Therapeutic Data Commons. 47 ChEMBL refers to 30 data sets curated from the ChEMBL database 62 by van Tilborg et al 11 The relative RMSE is the average, normalized RMSE obtained from 5-fold cross validation. Details of the fingerprints and descriptors used as molecular representations can be found in Methods.…”

“…Three sets of regression tasks were used in this work. Structure−property landscapes related to regression tasks were retrieved from the Therapeutic Data Commons (TDC) 47 using the Python library PyTDC (v. 0.3.6) and from the previous work of van Tilborg et al 11 A total of 55 regression data sets, split across three groups, were considered.…”

“…49 (2) The second group of data sets, referred to as "TDC", comprised 12 data sets that featured pharmaco-kinetic and toxicological properties and were obtained from the TDC: Caco2_Wang, 52 Lipophilicity_AstraZeneca, 53,54 Solubili-ty_AqSolDB, 5 5 HydrationFreeEnergy_FreeSolv, 5 6 PPBR_AZ, 54 VDss_Lombardo, 57 Half_Life_Obach, 58 Clear-ance_Hepatocyte_AZ, 54,59 Clearance_Microsome_AZ, 54,59 LD50_Zhu, 60 herg_central/hERG_at_1uM, 61 and herg_central/hERG_at_10uM. 61 (3) The third group of data sets, referred to as "ChEMBL", comprised 30 SAR data sets from ChEMBL 62 that were curated by van Tilborg et al 11 To reduce the computational cost of performing these tests, data set sizes were capped at 10,000 molecules; data sets containing a larger number of entries were subsampled at random (using a fixed seed for reproducibility).…”

DeepAC – conditional transformer-based chemical language model for the prediction of activity cliffs formed by bioactive compounds

“…8,9 ACs were rst accurately predicted using support vector machine (SVM) modeling on the basis of special kernel functions enabling compound pair predictions. 9 These ndings have also catalyzed further AC predictions using SVR variants [10][11][12] and other methods, [13][14][15][16][17][18] as discussed below. Recently, various deep neural network architectures have been used to predict ACs from images 14,15 and molecular graphs using representation learning 16 or derive regression models for potency prediction of AC compounds.…”

“…Recently, various deep neural network architectures have been used to predict ACs from images 14,15 and molecular graphs using representation learning 16 or derive regression models for potency prediction of AC compounds. 17,18 In this work, we further extend this methodological spectrum by introducing chemical language models for combined AC prediction and generative compound design. Compared to earlier studies predicting ACs using classication models, the approach presented herein was designed to extend AC predictions with the capacity to produce new AC compounds, thus integrating predictive and generative modeling in the context of AC analysis and AC-based compound design.…”