Characterizing Uncertainty in Machine Learning for Chemistry

Abstract

Characterizing uncertainty in machine learning models has recently gained interest in the context of machine learning reliability, robustness, safety, and active learning. Here, we separate the total uncertainty into contributions from noise in the data (aleatoric) and shortcomings of the model (epistemic), further dividing epistemic uncertainty into model bias and variance contributions. We systematically address the influence of noise, model bias, and model variance in the context of chemical property predictions, where the diverse nature of target properties and the vast chemical chemical space give rise to many different distinct sources of prediction error. We demonstrate that different sources of error can each be significant in different contexts and must be individually addressed during model development. Through controlled experiments on datasets of molecular properties, we show important trends in model performance associated with the level of noise in the dataset, size of the dataset, model architecture, molecule representation, ensemble size, and dataset splitting. In particular, we show that 1) noise in the test set can limit a model's observed performance when the actual performance is much better, 2) using size-extensive model aggregation structures is crucial for extensive property prediction, 3) ensembling is a reliable tool for uncertainty quantification and improvement specifically for the contribution of model variance, and 4) evaluations of cross-validation models understate their performance. We develop general guidelines on how to improve an underperforming model when falling into different uncertainty contexts.

Autonomous, multiproperty-driven molecular discovery: From predictions to measurements and back

“…For candidates with an experimental plan, their dye-like properties were predicted by using a set of Chemprop (14,15) property prediction models: wavelength of maximum absorption, partition coefficient, and photooxidative degradation rate. Chemprop is lightweight and fast, so an ensemble of models can be automatically retrained from the whole dataset with each batch of experimental data (supplementary materials, SM4) (14)(15)(16)41). The ensemble variance is used as a proxy for model uncertainty to further inform molecule selection.…”

“…This work focuses on organic small molecules that must manifest multiple distinct properties simultaneously, constraining molecular design and increasing structural complexity. Emerging predictive tools can quickly generate new candidate molecules (8)(9)(10)(11)(12)(13), predict the performance of candidates (11,(14)(15)(16), and propose practical reaction pathways (17-21); meanwhile, chemical automation can now dependably conduct experiments with minimal human intervention after an initial setup phase. Integrating generative algorithms, computer-aided synthesis planning (CASP), iteratively updated large datasets, and automated chemical synthesis, purification, and characterization for each step of the design-make-testanalyze (DMTA) cycle all into a single workflow could improve experiment efficiency and ultimately enable autonomous chemical discovery.…”

Comment on ‘Physics-based representations for machine learning properties of chemical reactions’

“…When creating training, validation, and testing sets, we use five folds, each with an 85:5:10 split. Our results report the mean and standard deviation of each fold to give some sense of model uncertainty; future work could more rigorously examine uncertainty estimation [159][160][161][162][163][164][165][166][167][168], though it is sometimes unclear which method is best [169]. We use both random splitting and scaffold splitting on the reactant SMILES, which partitions the data based on Bemis-Murcko scaffold splits [70], as calculated by RDKit [170].…”

Characterizing Uncertainty in Machine Learning for Chemistry

“…62 Other scripts necessary to train the models analyzed in this work and recreate the results are provided through GitHub (https://github.com/cjmcgill/ characterizing_uncertainty_scripts). 63…”