Uncertainty quantification in machine learning and nonlinear least squares regression models

Abstract

Machine learning (ML) models are valuable research tools for making accurate predictions. However, ML models often unreliably extrapolate outside their training data. The multiparameter delta method quantifies uncertainty for ML models (and generally for other nonlinear models) with parameters trained by least squares regression. The uncertainty measure requires the gradient of the model prediction and the Hessian of the loss function, both with respect to model parameters. Both the gradient and Hessian can be readily obtained from most ML software frameworks by automatic differentiation. We show examples of the uncertainty method in applications of molecular simulations and neural networks. We further show that the uncertainty measure is larger for input space regions that are not part of the training data. Therefore, this method can be used to identify extrapolation and to aid in selecting training data or assessing model reliability.

Model-Specific to Model-General Uncertainty for Physical Properties

“…The delta method follows previous work. 15 There is a difference between the uncertainty calculation for stabilized jellium (SJ), Anton−Schmidt (AS), and polynomial3 (P3), compared with the other models. For these models, the physical properties are derived quantities or predicted outputs, but for the other models, they are direct parameters.…”

“…The delta method scales as O(d 3 ) with d as number of model parameters and O(n) with n as number of data points, with research to improve scaling. 15 Bayesian regression is harder to set up. In our experience, pyro is generally easier to use than tensorflow-probability.…”

“…13,14 In previous work, we described some methods of obtaining uncertainty including ensembles, delta method, and Gaussian process, while focusing on the delta method. 15 Apart from Gaussian processes, the methods were specific to standard nonlinear regression. A point estimate of the parameters was obtained by minimizing the mean squared error, and the parameter and model prediction standard errors were obtained from the separate methods described.…”

“…Many of the analytical functions have the equilibrium energy, volume, and bulk modulus as parameters themselves, which makes their standard error available from the inverse Fisher information matrix. 15 For other functions, it is possible to calculate the uncertainty of the physical property under nonlinear regression and the delta method when the derivative of the property, with respect to the function parameters, exists.…”

Prediction uncertainty validation for computational chemists

“…Uncertainty quantification (UQ) is becoming a major issue for chemical machine learning (ML), 1 notably for the prediction of molecular and material properties. [2][3][4][5][6][7][8] This is also the case for quantum chemistry, when a level of confidence on predictions is sought out. 1,[9][10][11][12][13][14][15][16][17][18][19] In these contexts, the validation of UQ outputs is essential to enable their re-use in applications such as active learning or actionable predictions for the industry.…”