Theoretical characterization of uncertainty in high-dimensional linear classification

Abstract

Being able to reliably assess not only the accuracy but also the uncertainty of models' predictions is an important endeavour in modern machine learning. Even if the model generating the data and labels is known, computing the intrinsic uncertainty after learning the model from a limited number of samples amounts to sampling the corresponding posterior probability measure. Such sampling is computationally challenging in high-dimensional problems and theoretical results on heuristic uncertainty estimators in high-dimensions are thus scarce. In this manuscript, we characterise uncertainty for learning from limited number of samples of high-dimensional Gaussian input data and labels generated by the probit model. In this setting, the Bayesian uncertainty (i.e. the posterior marginals) can be asymptotically obtained by the approximate message passing algorithm, bypassing the canonical but costly Monte Carlo sampling of the posterior.
We then provide a closed-form formula for the joint statistics between the logistic classifier, the uncertainty of the statistically optimal Bayesian classifier and the ground-truth probit uncertainty. The formula allows us to investigate calibration of the logistic classifier learning from limited amount of samples. We discuss how over-confidence can be mitigated by appropriately regularising.

Calibration of uncertainty in the active learning of machine learning force fields

“…It appears that a model becomes increasingly underconfident as the training set grows, hence the increase in miscalibration area. This is a surprising result as it has been shown that a well-specified model should be calibrated in the limit where the training set size is much larger than the number of features [42]. Nonetheless, similar behavior has been reported with GP-deep neural network hybrids [43], and overparameterized deep neural networks [44] in which calibration error increases with training set size and training time, respectively.…”

Uncertainty quantification by direct propagation of shallow ensembles

“…In all cases in which an uncertainty-quantified models is miscalibrated (e.g. for ensembles of neural networks that tend to produce overconfident uncertainty estimates [68,69,71,72]), it is possible to apply a post-hoc calibration step on a hold-out set (in practice we use the validation set) to globally correct a model's uncertainty estimates. In the simplest case, assuming that the target properties are Gaussian distributed, one can apply a simple rescaling σ ← ασ.…”