The Impact of Protein Structure and Sequence Similarity on the Accuracy of Machine-Learning Scoring Functions for Binding Affinity Prediction

Abstract

It has recently been claimed that the outstanding performance of machine-learning scoring functions (SFs) is exclusively due to the presence of training complexes with highly similar proteins to those in the test set. Here, we revisit this question using 24 similarity-based training sets, a widely used test set, and four SFs. Three of these SFs employ machine learning instead of the classical linear regression approach of the fourth SF (X-Score which has the best test set performance out of 16 classical SFs). We have found that random forest (RF)-based RF-Score-v3 outperforms X-Score even when 68% of the most similar proteins are removed from the training set. In addition, unlike X-Score, RF-Score-v3 is able to keep learning with an increasing training set size, becoming substantially more predictive than X-Score when the full 1105 complexes are used for training. These results show that machine-learning SFs owe a substantial part of their performance to training on complexes with dissimilar proteins to those in the test set, against what has been previously concluded using the same data. Given that a growing amount of structural and interaction data will be available from academic and industrial sources, this performance gap between machine-learning SFs and classical SFs is expected to enlarge in the future.

From machine learning to deep learning: Advances in scoring functions for protein–ligand docking

“…The final assessment on the PDBbind benchmark showed that this version could yield a considerable scoring power as v2 ( R p = .803). Besides, a recent study indicated that RF‐Score‐v3 outperformed X‐Score even when 68% of the most similar proteins were removed from the training set, which further verified its superiority …”

“…However, Ballester, the author of RF‐Score, doubted that that LCOCV might be ultimately of little practical value, because these similarities could not be ignored when a SF was employed in a real scenario . In addition, by conducting a similar study as Li et al did, they concluded that ML‐based SFs could also learn from dissimilar training complexes, because RF‐Score was able to outperform X‐Score even when it was trained based on just 32% of the most dissimilar complexes. Thus, in our opinion, in order to develop a reliable SF, both the conventional random training/test splitting and LCOCV are needed, so that a comparison can be made to gain a deeper understanding of the impact of the SF itself and the composition of the dataset on the performance of a SF.…”

Selecting machine-learning scoring functions for structure-based virtual screening

“…It is important to note too that PM decoys are not required either to train or test QSAR models [35], despite predicting exactly the same in vitro potency/affinity endpoints as SFs (e.g. K d is predicted by both SFs [36,37] and QSAR models [38,39]).…”

CBSF: A New Empirical Scoring Function for Docking Parameterized by Weights of Neural Network

“…Therefore, the accuracy of our scoring function is comparable or higher than for most other known methods, both classical (empirical, force-eld and knowledge-based, R ∼ 0.4-0.7(Y. Li et al, 2014;Su et al, 2019)) and machine-learning based (R ∼ 0.6-0.8)(H. Li et al, 2018;Y. Li et al, 2014;Su et al, 2019).…”