Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout

Abstract

While the use of deep learning in drug discovery is gaining increasing attention, the lack of methods to computate reliable errors in prediction for Neural Networks prevents their application to guide decision making in domains where identifying unreliable predictions is essential, e.g. precision medicine. Here, we present a framework to compute reliable errors in prediction for Neural Networks using Test-Time Dropout and Conformal Prediction. Specifically, the algorithm consists of training a single Neural Network using dropout, and then applying it N times to both the validation and test sets, also employing dropout in this step. Therefore, for each instance in the validation and test sets an ensemble of predictions were generated. The residuals and absolute errors in prediction for the validation set were then used to compute prediction errors for test set instances using Conformal Prediction. We show using 24 bioactivity data sets from ChEMBL 23 that dropout Conformal Predictors are valid (i.e., the fraction of instances whose true value lies within the predicted interval strongly correlates with the confidence level) and efficient, as the predicted confidence intervals span a narrower set of values than those computed with Conformal Predictors generated using Random Forest (RF) models. Lastly, we show in retrospective virtual screening experiments that dropout and RF-based Conformal Predictors lead to comparable retrieval rates of active compounds. Overall, we propose a computationally efficient framework (as only N extra forward passes are required in addition to training a single network) to harness Test-Time Dropout and the Conformal Prediction framework, and to thereby generate reliable prediction errors for deep Neural Networks. Machine Learning -Data SplittingThe data sets were randomly split into a training set (70% of the data), a validation set (15%), and a test set (15%). For each data set, the training set was used to train a given network, whereas the validation set served to monitor the performance of the network during the training phase. In case of RF models, both the training and validation sets were used for model training.The predictive power of the final RF and DNN model was evaluated on the test set. The above split (and associated model training and testing) was repeated 20 times with random data set assignments. -Deep Neural Networks (DNN)DNNs were trained using the python library Pytorch 48 . We defined four hidden layers, composed of 1000, 1000, 100 and 10 nodes, respectively. The number of neurons in each layer was selected to be smaller than the input fingerprint size to reduce the chances of overfitting 49 .Rectified linear unit (ReLU) activation was used in all cases. The training data was processed in batches of size equal to 15% of the number of instances. We used Stochastic Gradient Descent with Nesterov momentum, which was set to 0.9 and kept constant during the training phase 50 .The networks were trained over 4,000 epochs, and early stopping was used in all cases, i.e., the ...

An Analysis of Proteochemometric and Conformal Prediction Machine Learning Protein-Ligand Binding Affinity Models

“…The scaled pIC50 MSE values in Table 1 translate to a root mean squared error of approximately 0.8 pIC50 units. These values are in the range of expected errors for ML models trained on heterogeneous ChEMBL25 data (Kalliokoski et al, 2013 ), and are in agreement with prior literature that also demonstrated that RF and FFN models approached the upper limit of overall accuracy across the dataset, given the heterogeneous IC50 measurements in ChEMBL25 (Cortés-Ciriano and Bender, 2019a ).…”

“…Interestingly, there is very little variation in the size of the FFN confidence intervals across all predictions on the validation set ( Figure 4A ) but this is still sufficient for the FFN to generate valid prediction intervals ( Figure 4B ). In total, conformal prediction is able to accurately gauge both RF and FFN model confidence for predictions on held-out validation data, in agreement with prior literature (Svensson et al, 2018 ; Cortés-Ciriano and Bender, 2019a ).…”

“…Conformal prediction requires a way to gauge the similarity of a new piece of data to training data. Recent literature has shown that the standard deviation across the trees of a RF model (Svensson et al, 2018 ), and the use of test-time dropout in the case of FFN models (Cortés-Ciriano and Bender, 2019a ), provide valid and efficient conformal predictors. However, these methods have not been analyzed in prospective settings extensively.…”

QSAR-derived affinity fingerprints (part 2): modeling performance for potency prediction

“…Further information about the data sets is given in Table 1 and in a previous study by the authors [42]. We also collected 25 QSAR data sets for validation from previous work by the authors (Table 2) [42][43][44]. All data sets used in this study, as well as the code required to generate the results presented herein, are publicly available at https ://githu b.com/isidr oc/QAFFP _regre ssion .…”

Bioactivity descriptors for uncharacterized compounds

“…Having trained and validated the signaturizers, we massively inferred missing signatures for the ~800,000 molecules available in the CC, obtaining a complete set of 25x128-dimensional signatures for each molecule (chemicalchecker.org/downloads). To explore the reliability of the inferred signatures, we assigned an 'applicability' score (α) to predictions based on the following: (a) the proximity of a predicted signature to true (experimental) signatures available in the training set; (b) the robustness of the SNN output to a test-time data dropout 10 ; and (c) the accuracy expected a priori based on the experimental CC datasets available for the molecule (Figure 2a).…”