Characterizing Uncertainty in Machine Learning for Chemistry

Abstract

Characterizing uncertainty in machine learning models has recently gained interest in the context of machine learning reliability, robustness, safety, and active learning. Here, we separate the total uncertainty into contributions from noise in the data (aleatoric) and shortcomings of the model (epistemic), further dividing epistemic uncertainty into model bias and variance contributions. We systematically address the influence of noise, model bias, and model variance in the context of chemical property predictions, where the diverse nature of target properties and the vast chemical chemical space give rise to many different distinct sources of prediction error. We demonstrate that different sources of error can each be significant in different contexts and must be individually addressed during model development. Through controlled experiments on data sets of molecular properties, we show important trends in model performance associated with the level of noise in the data set, size of the data set, model architecture, molecule representation, ensemble size, and data set splitting. In particular, we show that 1) noise in the test set can limit a model’s observed performance when the actual performance is much better, 2) using size-extensive model aggregation structures is crucial for extensive property prediction, and 3) ensembling is a reliable tool for uncertainty quantification and improvement specifically for the contribution of model variance. We develop general guidelines on how to improve an underperforming model when falling into different uncertainty contexts.

Compositional Deep Probabilistic Models of DNA-Encoded Libraries

“…Random splits are not always ideal for testing molecule data sets. , In order to test the generalizability of molecule representations, many approaches attempt to split molecule by molecular scaffolds . For DELs, rather than generic molecular scaffolding strategies, synthons provide a natural grouping and separation of the chemical space.…”

Machine learning-based correction for spin–orbit coupling effects in NMR chemical shift calculations

“…Note that the performance of an ML approach is directly influenced and limited by the quality – especially the noisiness – of the reference data. 64 PBE0 is a generally robust functional that usually yields good NMR properties and especially the SO-relativistic variant has proven reliable performance in our previous studies on 29 Si 10 and 119 Sn 11 NMR chemical shifts. Furthermore, in contrast to full four-component relativistic methods, SO-ZORA-PBE0 is still feasible for the medium-sized (>40 atoms) molecules included in the data set.…”

“…The information included as input is of central importance for the quality and performance of the ML model. 64 In the case of Δ SO -ML, the input feature vector is constructed such that it contains information about the geometric (solely from the three-dimensional structure), electronic (from the converged density matrix of the DFT single-point calculation), and magnetic (from the DFT NMR shielding constant calculation) surrounding of each atom of interest. The majority of the descriptors of these categories was taken from the Δ corr -ML model and some were omitted.…”

Subgraph Isomorphic Decision Tree to Predict Radical Thermochemistry with Bounded Uncertainty Estimation

“…There are two sources of uncertainty in machine learning for chemistry: the aleatoric uncertainty and the epistemic uncertainty. 49 The aleatoric uncertainty is associated with noise in the training data and is not reducible during training, while the epistemic uncertainty is associated with the model bias and variances and is reducible. The original SIDT algorithm 26 extends the tree until exhaustion.…”