Tapping on the Black Box: How Is the Scoring Power of a Machine-Learning Scoring Function Dependent on the Training Set?

Abstract

In recent years, protein−ligand interaction scoring functions derived through machine-learning are repeatedly reported to outperform conventional scoring functions. However, several published studies have questioned that the superior performance of machine-learning scoring functions is dependent on the overlap between the training set and the test set. In order to examine the true power of machine-learning algorithms in scoring function formulation, we have conducted a systematic study of six off-the-shelf machine-learning algorithms, including Bayesian Ridge Regression (BRR), Decision Tree (DT), K-Nearest Neighbors (KNN), Multilayer Perceptron (MLP), Linear Support Vector Regression (L-SVR), and Random Forest (RF). Model scoring functions were derived with these machine-learning algorithms on various training sets selected from over 3700 protein−ligand complexes in the PDBbind refined set (version 2016). All resulting scoring functions were then applied to the CASF-2016 test set to validate their scoring power. In our first series of trial, the size of the training set was fixed; while the overall similarity between the training set and the test set was varied systematically. In our second series of trial, the overall similarity between the training set and the test set was fixed, while the size of the training set was varied. Our results indicate that the performance of those machinelearning models are more or less dependent on the contents or the size of the training set, where the RF model demonstrates the best learning capability. In contrast, the performance of three conventional scoring functions (i.e., ChemScore, ASP, and X-Score) is basically insensitive to the use of different training sets. Therefore, one has to consider not only "hard overlap" but also "soft overlap" between the training set and the test set in order to evaluate machine-learning scoring functions. In this spirit, we have complied data sets based on the PDBbind refined set by removing redundant samples under several similarity thresholds. Scoring functions developers are encouraged to employ them as standard training sets if they want to evaluate their new models on the CASF-2016 benchmark.

Learning from Docked Ligands: Ligand-Based Features Rescue Structure-Based Scoring Functions When Trained On Docked Poses

“…All three models (LB, SB, and HB) are strongly affected by the similarity between the training and test sets, with the exclusion of training set complexes with similar proteins or ligands to those in the test set significantly reducing performance. These results echo our earlier results 15 and those of Su et al 33 , indicating that even when potentially lessaccurate binding poses are used, it is necessary to consider the effect of biases in the available structural data when training and evaluating models. The inclusion of ligand-based features in structure-based models always improves performance when using docked poses, and only ceases to improve performance when using crystal poses if the maximum fingerprint Tanimoto similarity between ligands in the training and test set is less than or equal to 0.5.…”

RASPD+: Fast Protein-Ligand Binding Free Energy Prediction Using Simplified Physicochemical Features

“…Random forest methods, which were best suited for the G regression on known binders, were for most proteins outperformed by the simpler linear regression methods. This observation might support the recent finding that random forest methods, in particular, benefit from highly similar training molecules (Su et al, 2020). Considering the strengths and weaknesses of the different machine learning methods, we therefore recommend that for applications of RASPD+, the results of the seven different machine learning methods are combined by picking top candidates from the rankings produced by each method.…”

“…By splitting the PDBbind training, testing, and validation data in a nested cross-validation setup, we were able to assess reliably that random forest models, particularly the extremely random forest model, performed best on this type of data. While this splitting strategy increases confidence in the comparison of learning methods and feature importance analysis within the study, other data set splitting strategies, which explicitly control how similar proteins or ligands are between training and test sets (Feinberg et al, 2018;Sieg et al, 2019;Su et al, 2020), may be more appropriate to assess performance on completely different ligands or proteins directly.…”

RASPD+: Fast Protein-Ligand Binding Free Energy Prediction Using Simplified Physicochemical Features

“…The highest enrichment was observed for ion channels, G-protein coupled receptors (GPCR), and kinases in both settings (Table 4). analysis within the study, other data set splitting strategies, which explicitly control how similar proteins or ligands are between training and test sets 14,52,53 , may be more appropriate to assess performance on completely different ligands or proteins directly.…”