The brittleness of deep learning models is ailing their deployment in real-world applications, such as transportation and airport security. Most work focuses on developing accurate models that only deliver point estimates without further information on model uncertainty or confidence. Ideally, a learning model should compute the posterior predictive distribution, which contains all information about the model output. We cast the problem of density tracking in neural networks using Particle Filtering, a powerful class of numerical methods for the solution of optimal estimation problems in non-linear, non-Gaussian systems. Particle filters are a powerful alternative to Markov chain Monte Carlo algorithms and enjoy established convergence and performance guarantees. In this paper, we advance a particle filtering framework for neural networks, where the predictive output is a distribution. The mean of this distribution serves as the point estimate decision and its variance provides the model confidence in the decision. Our framework shows increased robustness under noisy conditions. Additionally, the predictive variance increases monotonically with decreasing signal-to-noise ratio (SNR); thus reflecting a lower confidence or higher uncertainty. This paper serves as a pioneering proofof-concept framework that will allow the development of a theoretical understanding of robust neural networks.

Machine learning models have achieved human-level performance on various tasks. This success comes at a high cost of computation and storage overhead, which makes machine learning algorithms difficult to deploy on edge devices. Typically, one has to partially sacrifice accuracy in favor of an increased performance quantified in terms of reduced memory usage and energy consumption. Current methods compress the networks by reducing the precision of the parameters or by eliminating redundant ones. In this paper, we propose a new insight into network compression through the Bayesian framework. We show that Bayesian neural networks automatically discover redundancy in model parameters, thus enabling self-compression, which is linked to the propagation of uncertainty through the layers of the network. Our experimental results show that the network architecture can be successfully compressed by deleting parameters identified by the network itself while retaining the same level of accuracy.

This is a repository copy of Robust learning via ensemble density propagation in deep neural networks.

No abstract

No abstract