Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction

Abstract

Advances in deep neural network (DNN) based molecular property prediction have recently led to the development of models of remarkable accuracy and generalization ability, with graph convolution neural networks (GCNNs) reporting state-of-the-art performance for this task. However, some challenges remain and one of the most important that needs to be fully addressed concerns uncertainty quantification. DNN performance is affected by the volume and the quality of the training samples. Therefore, establishing when and to what extent a prediction can be considered reliable is just as important as outputting accurate predictions, especially when out-of-domain molecules are targeted. Recently, several methods to 1 arXiv:1910.03127v1 [cs.LG] 7 Oct 2019 account for uncertainty in DNNs have been proposed, most of which are based on approximate Bayesian inference. Among these, only a few scale to the large datasets required in applications. Evaluating and comparing these methods has recently attracted great interest, but results are generally fragmented and absent for molecular property prediction. In this paper, we aim to quantitatively compare scalable techniques for uncertainty estimation in GCNNs. We introduce a set of quantitative criteria to capture different uncertainty aspects, and then use these criteria to compare MC-Dropout, deep ensembles, and bootstrapping, both theoretically in a unified framework that separates aleatoric/epistemic uncertainty and experimentally on the QM9 dataset. Our experiments quantify the performance of the different uncertainty estimation methods and their impact on uncertainty-related error reduction. Our findings indicate that ensembling and bootstrapping consistently outperform MC-Dropout, with different context-specific pros and cons. Our analysis also leads to a better understanding of the role of aleatoric/epistemic uncertainty and highlights the challenge posed by out-of-domain uncertainty.

Methods for comparing uncertainty quantifications for material property predictions

“…Thus a higher C v may indicate more robust uncertainty estimates. But as Scalia et al 16 point out, the optimal dispersion is a function of the validation/test data distribution. Therefore, C v should be used as a secondary screening metric rather than a primary performance metric.…”

Exchange Spin Coupling from Gaussian Process Regression

“…When making predictions with ML for molecular structures outside of the training set, it is desirable to have a measure for how reliable those predictions are, i.e., an error or uncertainty estimate. Such estimates have been implemented in chemistry-related ML frameworks 73 , e.g. based on a latent space distance metric for artificial neural networks 74 or employing resampling techniques such as bootstrapping or subsampling 75 .…”

DeepGraphMolGen, a multi-objective, computational strategy for generating molecules with desirable properties: a graph convolution and reinforcement learning approach

“…Also, for generating desired molecules, the QSAR models need to be accurate and robust in order to evaluate accurately the property of the generated molecules. Recent works such as [96] include uncertainty metrics for property discrimination, and benchmarking models are also available [97]. In conclusion, we here add to the list of useful, generative molecular methods for virtual screening by combining molecular graph encoding, reinforcement learning and multi-objective optimisation within a single strategy.…”