Hidden Bias in the DUD-E Dataset Leads to Misleading Performance of Deep Learning in Structure-Based Virtual Screening

Abstract

<p>Recently much effort has been invested in using convolutional neural network (CNN) models trained on 3D structural images of protein-ligand complexes to distinguish binding from non-binding ligands for virtual screening. However, the dearth of reliable protein-ligand x-ray structures and binding affinity data has required the use of constructed datasets for the training and evaluation of CNN molecular recognition models. Here, we outline various sources of bias in one such widely-used dataset, the Directory of Useful Decoys: Enhanced (DUD-E). We have constructed and performed tests to investigate whether CNN models developed using DUD-E are properly learning the underlying physics of molecular recognition, as intended, or are instead learning biases inherent in the dataset itself. We find that superior enrichment efficiency in CNN models can be attributed to the analogue and decoy bias hidden in the DUD-E dataset rather than successful generalization of the pattern of protein-ligand interactions. Comparing additional deep learning models trained on PDBbind datasets, we found that their enrichment performances using DUD-E are not superior to the performance of the docking program AutoDock Vina. Together, these results suggest that biases that could be present in constructed datasets should be thoroughly evaluated before applying them to machine learning based methodology development. </p>

Generating Property-Matched Decoy Molecules Using Deep Learning

“…We believe that this substantial reduction in bias will benefit the development and improve generalisation of structurebased virtual screening methods. Currently, methods can perform well on retrospective benchmarks without performing molecular recognition by simply learning underlying biases (3,22,27). Thus it is unclear if improvements are genuine or due to more closely capturing these biases.…”

“…The reported results show that these methods substantially outperform other methodologies such as empirical and knowledgebased scoring functions at SBVS. Concerningly, some reports have suggested that a key driver of the performance of machine learning-based systems is hidden biases in the training data, such as physicochemical differences, and that these methods are not learning to perform molecular recognition (3,22). Better decoy molecules are essential to remove the biases in datasets that are hindering the development of virtual screening methods.…”

DEEPScreen: high performance drug–target interaction prediction with convolutional neural networks using 2-D structural compound representations

“…Also, 3-D structural information (especially the target-ligand complexes) is only available for a small portion of the DTI space; as a result, their coverage is comparably low and they generally are not suitable for large-scale DTI prediction. It is also important to note that the DUD-E benchmark dataset is reported to suffer from negative selection bias problem, 43 and thus, the results based on this dataset may not be conclusive.…”

“…Another risk is the negative selection bias, where negative samples (i.e., inactive or non-binder compounds) in the training and/or test datasets are structurally similar to each other in a way, which is completely unrelated to their binding related properties. 43 So, a machine learning classier can easily exploit this feature to successfully separate them from the positives. Both of these cases would result in an overestimation of the model performance during benchmarks, especially when the tests are made to infer to performance of the models in predicting completely novel binders to the modelled target proteins.…”

“…In DUD-E, most of the property matched decoys (i.e., negatives) were found to be highly biased, as the models trained on specic targets were highly successful in identifying the negatives of completely different targets. 43 In other words, most of the decoys shared features that make them non-binders to nearly all target proteins, and care should be taken while evaluating predictive models on this benchmark. In this study, we evaluated the performance of DEEPScreen on 5 different datasets (e.g., large-scale random-split dataset, both chemical and negative selection bias free representative target dataset, ChEMBL temporal/time split dataset, MUV and DUD-E) in order to observe the behaviour of the system and its comparison with the state-of-the-art on benchmarks with differing strengths and weaknesses.…”