Experimental Error, Kurtosis, Activity Cliffs, and Methodology: What Limits the Predictivity of Quantitative Structure–Activity Relationship Models?

Abstract

Given a particular descriptor/method combination, some quantitative structure–activity relationship (QSAR) datasets are very predictive by random-split cross-validation while others are not. Recent literature in modelability suggests that the limiting issue for predictivity is in the data, not the QSAR methodology, and the limits are due to activity cliffs. Here, we investigate, on in-house data, the relative usefulness of experimental error, distribution of the activities, and activity cliff metrics in determining how predictive a dataset is likely to be. We include unmodified in-house datasets, datasets that should be perfectly predictive based only on the chemical structure, datasets where the distribution of activities is manipulated, and datasets that include a known amount of added noise. We find that activity cliff metrics determine predictivity better than the other metrics we investigated, whatever the type of dataset, consistent with the modelability literature. However, such metrics cannot distinguish real activity cliffs due to large uncertainties in the activities. We also show that a number of modern QSAR methods, and some alternative descriptors, are equally bad at predicting the activities of compounds on activity cliffs, consistent with the assumptions behind “modelability.” Finally, we relate time-split predictivity with random-split predictivity and show that different coverages of chemical space are at least as important as uncertainty in activity and/or activity cliffs in limiting predictivity.

Deep Learning Approaches in Predicting ADMET Properties

“…In general, machine learning approaches perform better with multiple experimental observations and molecular diversity. Repeated observations are useful for quantifying experimental variability in the assay and therefore the limits of predictability [1,17]. Molecular diversity in the training data allows models to generalize to a wide range of molecular scaffolds (global diversity) as well as learn nuances from smaller functional group perturbations (localized diversity).…”

Best practices for constructing, preparing, and evaluating protein-ligand binding affinity benchmarks

“…Publications with essential experimental controls reported -such as incubation time and concentration regime to demonstrate equilibrium -can add confidence to the reported affinity, however these may be performed and not reported [104]. Meta-analyses of both repeatability [105] and reproducibility [103] found errors in pKi of 0.3-0.4 log units (0.43-0.58 kcal mol -1 ) and 0.44 log units (0.64 kcal mol -1 ) respectively. Another analysis for reproducibility found that variability in pIC 50 were even 21-26% higher than for pKi data (0.55 log units) [100].…”

“…where R 2 max is the highest achievable R 2 for a dataset with a standard deviation of affinities (σ(affinity)) and an experimental uncertainty of σ(measurement error) [105]. This relation is illustrated in Figure 7.…”

Advances in exploring activity cliffs

“…This approach enabled the prediction of ACs of varying magnitude. However, as mentioned above, potency predictions for AC compounds using QSAR approaches are generally difficult, regardless of descriptors and methods used [40]. This is the case because QSAR modeling is principally based on the presence of SAR continuity when gradual changes in molecular structure cause small to moderate changes in potency.…”