Calibrated Prediction Intervals for Neural Network Regressors

Abstract

Ongoing developments in neural network models are continually advancing the state of the art in terms of system accuracy. However, the predicted labels should not be regarded as the only core output; also important is a well-calibrated estimate of the prediction uncertainty. Such estimates and their calibration are critical in many practical applications. Despite their obvious aforementioned advantage in relation to accuracy, contemporary neural networks can, generally, be regarded as poorly calibrated and as such do not produce reliable output probability estimates. Further, while post-processing calibration solutions can be found in the relevant literature, these tend to be for systems performing classification. In this regard, we herein present two novel methods for acquiring calibrated predictions intervals for neural network regressors: empirical calibration and temperature scaling. In experiments using different regression tasks from the audio and computer vision domains, we find that both our proposed methods are indeed capable of producing calibrated prediction intervals for neural network regressors with any desired confidence level, a finding that is consistent across all datasets and neural network architectures we experimented with. In addition, we derive an additional practical recommendation for producing more accurate calibrated prediction intervals. We release the source code implementing our proposed methods for computing calibrated predicted intervals. The code for computing calibrated predicted intervals is publicly available 1 .

Considerations for a More Ethical Approach to Data in AI: On Data Representation and Infrastructure

“…Given the “black-box” nature of deep learning, there have been numerous approaches to quantifying confidence (Kendall and Cipolla, 2016 ; Keren et al, 2018 ). One popular procedure for measuring confidence is the Monte Carlo dropout .…”

“…In AI data, the terms confidence and trust are applied to ensure reliability, i.e., having confidence in the data results in deeper trust (Arnold et al, 2019 ). In this context, trust is a qualitative term, and although confidence can fall into these interpretations relating to enhanced moral understanding (Blass, 2018 ), the term confidence typically refers to a quantifiable measure to base trust on (Zhang et al, 2001 ; Keren et al, 2018 ).…”

Estimating Expected Calibration Errors

“…For each mode we build the mean vector with elements uniformly drawn in [0, 1], and the covariance matrix is built as follows: we first sample a matrix with elements uniformly drawn in [−0.3, 0.3] then multiply it with its transposition to get the required positive definite matrix. This sample set generation is produced with various number of classes (2,5,7) and dimensions of the feature space (2, 5, 7) with 5 different large datasets sampled from each combination, resulting into 45 synthetic distributions.…”

“…More recently [7] made clearer the notion of calibration for multiclass classifiers, and new estimators of the ECE with adaptive binning have been proposed in [12] along side with uncertainty aware reliability diagrams [1]. Although the notion of calibration was originally defined for classifiers, this notion is currently being generalized to regression [5,15].…”

A Deep Learning Integrated Lee–Carter Model

“…The literature on this topic is not extensive and the question is still open. Some scholars (Khosravi et al (2011), Keren et al (2019) have faced this problem, but none of them has so far treated the question concerning NNs (and more generally deep learning) applied to time series. However, the LSTM network demonstrates being a good candidate to meet the need of predicting the mortality trend over time more accurately.…”