Latent Biases in Machine Learning Models for Predicting Binding Affinities Using Popular Data Sets

Abstract

Drug design involves the process of identifying and designing molecules that bind well to a given receptor. A vital computational component of this process is the protein−ligand interaction scoring functions that evaluate the binding ability of various molecules or ligands with a given protein receptor binding pocket reasonably accurately. With the publicly available protein− ligand binding affinity data sets in both sequential and structural forms, machine learning methods have gained traction as a top choice for developing such scoring functions. While the performance shown by these models is optimistic, there are several hidden biases present in these data sets themselves that affect the utility of such models for practical purposes such as virtual screening. In this work, we use published methods to systematically investigate several such factors or biases present in these data sets. In our analysis, we highlight the importance of considering sequence, protein−ligand interaction, and pocket structure similarity while constructing data splits and provide an explanation for good protein-only and ligand-only performances in some data sets. Through this study, we provide to the community several pointers for the design of binding affinity predictors and data sets for reliable applicability.

Toward generalizable structure‐based deep learning models for protein–ligand interaction prediction: Challenges and strategies

“…In chemistry, random splitting has been shown to possess limitations in confirming the data generalizability 92,93 . Specifically, in the context of predicting PLI, where proteins and ligands and their associated relationships significantly affect results, using random splitting for data construction can introduce severe biases 57,94 . In the case of a refined (general)‐core splitting strategy, a model is trained with a refined or general set of PDBbind and tested on a high‐quality core set.…”

“…92,93 Specifically, in the context of predicting PLI, where proteins and ligands and their associated relationships significantly affect results, using random splitting for data construction can introduce severe biases. 57,94 In the case of a refined (general)-core splitting strategy, a model is trained with a refined or general set of PDBbind and tested on a high-quality core set. All models that used the PDBbind dataset for training and the CASF-2016 benchmark dataset 20 for testing correspond to the refined-core splitting.…”

Exploring protein–ligand binding affinity prediction with electron density-based geometric deep learning

“…16 Furthermore, existing datasets such as PDBbind 17,18 commonly used for model development have been criticized as biased because similar prediction accuracies were achieved irrespective of whether the whole protein-ligand complex, only the protein, or only the ligand was considered. 2,19,20 This potential bias in dataset composition makes it challenging to meaningfully assess and compare model performance. 2,16,19,20 As a potential remedy to the challenge of learning underlying dataset biases rather than meaningfully capturing physical interactions, Rognan and coworkers suggested to consider "only noncovalent interactions while omitting their protein and ligand atomic environments".…”

“…2,19,20 This potential bias in dataset composition makes it challenging to meaningfully assess and compare model performance. 2,16,19,20 As a potential remedy to the challenge of learning underlying dataset biases rather than meaningfully capturing physical interactions, Rognan and coworkers suggested to consider "only noncovalent interactions while omitting their protein and ligand atomic environments". 2 In addition to geometric deep learning approaches, 5,23-26 also simulation-based approaches (e.g., free energy perturbation, 27,28 MM/PBSA [29][30][31] ) are frequently used for binding affinity prediction.…”

“…2,19,20 This potential bias in dataset composition makes it challenging to meaningfully assess and compare model performance. 2,16,19,20 As a potential remedy to the challenge of learning underlying dataset biases rather than meaningfully capturing physical interactions, Rognan and coworkers suggested to consider “only noncovalent interactions while omitting their protein and ligand atomic environments”. 2…”