Exposing the Limitations of Molecular Machine Learning with Activity Cliffs

Abstract

Machine learning has become a crucial tool in drug discovery and chemistry at large, e.g., to predict molecular properties, such as bioactivity, with high accuracy. However, activity cliffs�pairs of molecules that are highly similar in their structure but exhibit large differences in potency�have received limited attention for their effect on model performance. Not only are these edge cases informative for molecule discovery and optimization but also models that are well equipped to accurately predict the potency of activity cliffs have increased potential for prospective applications. Our work aims to fill the current knowledge gap on best-practice machine learning methods in the presence of activity cliffs. We benchmarked a total of 24 machine and deep learning approaches on curated bioactivity data from 30 macromolecular targets for their performance on activity cliff compounds. While all methods struggled in the presence of activity cliffs, machine learning approaches based on molecular descriptors outperformed more complex deep learning methods. Our findings highlight large case-by-case differences in performance, advocating for (a) the inclusion of dedicated "activity-cliff-centered" metrics during model development and evaluation and (b) the development of novel algorithms to better predict the properties of activity cliffs. To this end, the methods, metrics, and results of this study have been encapsulated into an open-access benchmarking platform named MoleculeACE (Activity Cliff Estimation, available on GitHub at: https://github.com/molML/MoleculeACE). MoleculeACE is designed to steer the community toward addressing the pressing but overlooked limitation of molecular machine learning models posed by activity cliffs.

A Data-Driven Workflow for Assigning and Predicting Generality in Asymmetric Catalysis

“…This can be demonstrated by the number of unique catalysts present in the database that correspond to a particular chemotype. We recorded 27 urea-based catalysts that were good for at least one Mannich reaction which is greater than the number of secondary amines (21), although more reactions have been performed with secondary amines. Similarly, the number of unique cinchona alkaloids ( 16) is high despite a much lower number of reactions reported.…”

Predicting compound activity from phenotypic profiles and chemical structures

“…This approach has been used to synthesize and test promising compounds, and has resulted in novel therapeutics, including the discovery of a new antibiotic 5 . Many recent advances leverage deep learning formulations [6][7][8][9][10][11][12][13][14][15][16][17][18][19] , but there are still open challenges to realize the full potential of molecular property prediction, including data sparsity and imbalance 17 , and activity cliffs 20 among others. Using only chemical structures might have other limitations due to lack of information on biological contexts or how living organisms respond to treatments.…”

A Perspective on Explanations of Molecular Prediction Models

“…Matched molecular pairs (MMPs) are pairs of molecules that differ structurally at only one site by a known transformation. , MMPs are widely used in drug discovery and medicinal chemistry as these facilitate fast and easy understanding of structure–activity relationships. − Counterfactuals and MMP examples intersect if the structural change is associated with a significant change in the properties. If the associated changes in the properties are nonsignificant, the two molecules are known as bioisosteres. , The connection between MMPs and adversarial training examples has been explored in ref . MMPs, which belong to the counterfactual category, are commonly used in outlier and activity cliff detection .…”

“…If the associated changes in the properties are nonsignificant, the two molecules are known as bioisosteres. 116 , 117 The connection between MMPs and adversarial training examples has been explored in ref ( 118 ). MMPs, which belong to the counterfactual category, are commonly used in outlier and activity cliff detection.…”