Detection of outliers in classification by using quantified uncertainty in neural networks

Abstract

Neural Networks (NNs) can solve very hard classification and estimation tasks but are less well suited to solve complex sensor fusion challenges, such as end-to-end control of autonomous vehicles. Nevertheless, NN can still be a powerful tool for particular sub-problems in sensor fusion. This would require a reliable and quantifiable measure of the stochastic uncertainty in the predictions that can be compared to classical sensor measurements. However, current NN's output some figure of merit, that is only a relative model fit and not a stochastic uncertainty. We propose to embed the NN's in a proper stochastic system identification framework. In the training phase, the stochastic uncertainty of the parameters in the (last layers of the) NN is quantified. We show that this can be done recursively with very few extra computations. In the classification phase, Monte-Carlo (MC) samples are used to generate a set of classifier outputs. From this set, a distribution of the classifier output is obtained, which represents a proper description of the stochastic uncertainty of the predictions. We also show how to use the calculated uncertainty for outlier detection by including an artificial outlier class. In this way, the NN fits a sensor fusion framework much better. We evaluate the approach on images of handwritten digits. The proposed method is shown to be on par with MC dropout, while having lower computational complexity, and the outlier detection almost completely eliminates false classifications.

Modeling of the tire-road friction using neural networks including quantification of the prediction uncertainty

“…Furthermore, the second derivative of the likelihood function approximates the inverse of the Fisher information matrix. In [29], it was shown that for a classification problem, the Fisher information is given by…”

“…The high computational complexity is particularly true for NN classifiers, which have a huge dimension of the parameter space. In [29], a remedy to this is presented, which reduces the dimension in the sampling to an M -dimensional space. In this paper, it is done using the so-called delta method [36,38].…”

“…The first category of methods is based on designing the structure of the NN such that it learns its own uncertainty [16][17][18][19]. The second category is based on creating an ensemble of predictions, from which the uncertainty in the prediction could be computed [20][21][22][23][24][25][26][27][28][29][30].…”

“…Another well-used approximation is the so-called Laplacian approximation of a BNN, where the curvature of the likelihood function gives the distribution of the parameters [30][31][32][33][34]. In particular, this paper will focus on the linearized Laplacian approximation (LLA) [29,35], where the so-called delta method is used to model the distribution for the prediction rather than the distribution for the parameters [36][37][38].…”

“…Here, fusing the information is shown to lead to more robust decisions. Secondly, the delta method for the Laplacian approximation of BNN presented in [29] is extended such that the estimated probability can represent multimodal distributions. Thirdly, the presented methods demonstrate efficiency in detecting out-of-distribution examples.…”