Evaluation of Cross-Validation Strategies in Sequence-Based Binding Prediction Using Deep Learning

Abstract

Binding prediction between targets and drug-like compounds through Deep Neural Networks have generated promising results in recent years, outperforming traditional machine learning-based methods. However, the generalization capability of these classification models is still an issue to be addressed. In this work, we explored how di↵erent cross-validation strategies applied to data from di↵erent molecular databases a↵ect to the performance of binding prediction proteochemometrics models. These strategies are: (1) random splitting, (2) splitting based on K-means clustering (both 1 of actives and inactives), (3) splitting based on source database and (4) splitting based both in the clustering and in the source database. These schemas are applied to a Deep Learning proteochemometrics model and to a simple logistic regression model to be used as baseline. Additionally, two di↵erent ways of describing molecules in the model are tested: (1) by their SMILES and (2) by three fingerprints. The classification performance of our Deep Learning-based proteochemometrics model is comparable to the state of the art. Our results show that the lack of generalization of these models is due to a bias in public molecular databases and that a restrictive cross-validation schema based on compounds clustering leads to worse but more robust and credible results. Our results also show better performance when representing molecules by their fingerprints.

The effect of statistical normalisation on network propagation scores

“…Explanatory models have found use in the formal description of differences in performance as a function of design factors (Lopez-del Rio et al , 2019; Picart-Armada et al , 2019). Following (Picart-Armada et al , 2019), the trends in AUROC and AUPRC were described through logistic-like quasibinomial models with a logit link function, as a generalisation of logistic models to prevent over and under-dispersion issues.…”

Effect of Sequence Padding on the Performance of Protein-Based Deep Learning Models

“…CNNs imply translational invariance [10] and can be used to find relevant patterns with biological meaning [8,5,11,12]. For their part, bidirectional RNNs (and the derived Long Short-Term Memory and Gated Recurrent Units) are appropiate for modelling biological sequences since they are suited for data with a sequential but non-causal structure, variable length, and long-range dependencies [13,14,15,16]. Both architectures are usually combined, as in DEEPre [17], where a CNN-RNN model performs a hierarchical classification of enzymes.…”

“…The analogy between text and proteins, understood as sequences of characters with a meaning, motivates the application of Natural Language Processing (NLP) techniques to amino acid sequences. Along these lines, machine-learning derived embeddings [23,24,25,26] and one-hot encoding [14,9,27,12,17,7] have become very popular. Specifically, the latter method has been widely used in protein-based DL models since neural networks are able to extract features from raw data.…”

“…The main problem of one-hot encoding is that each protein has a different length, while all the input vectors should be of the same size to be fed into the model. To overcome this issue, sequence padding and truncation are usually applied [7,12,14,9]. This means establishing a common length for all the proteins and then, truncating longer proteins to that length or filling shorter proteins with an "artificial" character up until that length (see Figure 1A).…”

“…Padding zeros can be added at any point of the sequence, for example at the N-and C-terminals of the sequences [28]. In practice they are usually added at the end [7,14]. However, details on the concrete steps when padding the sequences are often omitted as they are deemed of low importance for the results of the study [12,9,27,17].…”