Measuring the Uncertainty of Predictions in Deep Neural Networks with Variational Inference

Abstract

We present a novel approach for training deep neural networks in a Bayesian way. Compared to other Bayesian deep learning formulations, our approach allows for quantifying the uncertainty in model parameters while only adding very few additional parameters to be optimized. The proposed approach uses variational inference to approximate the intractable a posteriori distribution on basis of a normal prior. By representing the a posteriori uncertainty of the network parameters per network layer and depending on the estimated parameter expectation values, only very few additional parameters need to be optimized compared to a non-Bayesian network. We compare our approach to classical deep learning, Bernoulli dropout and Bayes by Backprop using the MNIST dataset. Compared to classical deep learning, the test error is reduced by 15%. We also show that the uncertainty information obtained can be used to calculate credible intervals for the network prediction and to optimize network architecture for the dataset at hand. To illustrate that our approach also scales to large networks and input vector sizes, we apply it to the GoogLeNet architecture on a custom dataset, achieving an average accuracy of 0.92. Using 95% credible intervals, all but one wrong classification result can be detected.

Uncertainty Estimation in Deep Neural Networks for Point Cloud Segmentation in Factory Planning

“…One possibility of describing predictive uncertainty U pred in a label y belonging to an input x with weights w is based on entropy, i.e., U pred = H[y |w, x ] [31]. Another way of quantifying uncertainty in the network parameters of BNNs is presented in [7]. This approach only introduces two uncertainty parameters per network layer, which allows us to grasp uncertainty layer-wise, but it does not impair network convergence.…”

“…After having discussed the theoretical background, we describe our Bayesian model and the corresponding variational distribution. The model that we suggest has a similar structure to the framework in [7]. In the remaining section, the subscript indices w and b represent that a quantity is related to the network weights and biases, respectively.…”

“…For more detailed insights on the respective gradient updates, see [7]. We use the leaky ReLU activation function in the Bayesian model with a negative slope of 0.01 due to the dying ReLU problem.…”

“…The network is trained using variational inference with multivariate Gaussians with a diagonal covariance matrix as variational distribution. This approach hardly adds any additional parameters to be optimized during each backward pass [7]. Further, we formulate an approximate Bayesian neural network by applying dropout training, as suggested in [3].…”

From a Point Cloud to a Simulation Model—Bayesian Segmentation and Entropy Based Uncertainty Estimation for 3D Modelling

“…Further ways of uncertainty quantification include the variance of the predictive network outputs and the estimation of (95 %-)credible intervals [ 44 ]. In the former case the predictive variance is estimated empirically using the unbiased estimator for the variance.…”