Radial and Directional Posteriors for Bayesian Neural Networks

Abstract

We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widelyused Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via the radial density provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously.

Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement

“…Analogous to the likelihood function, variational dropout [37], which is used in this work, approximates the posteriors p(θ|D) by Gaussian distributions with diagonal covariance, imposing restrictive assumptions of unimodality and statistical independence between neural network weights. More recent advances in the Bayesian deep learning research [110][111][112][113][114][115] could be used to enhance the quality of parameter uncertainty estimation by allowing the model to capture multi-modality and statistical dependencies between parameters.…”

Priors in Bayesian Deep Learning: A Review

“…Another interesting prior is the radial-directional prior, which disentangles the direction of the weight vector from its length [168]. It is given by…”

“…where p dir is a distribution over the d-dimensional unit sphere and p rad is a distribution over R. It has been proposed by Oh et al [168] to use the von-Mises-Fisher distribution (see Eq. ( 13)) for p dir and the half-Cauchy (see Eq.…”

On The Distribution of Penultimate Activations of Classification Networks

“…As the vMF distribution is one of the simplest distributions for directional data, mixtures of vMFs have been widely used for clustering directional data [2,8]. For Bayesian inference of neural network weights, vMF distributions are used to model the directional statistics of the weights that are decomposed into radial and directional components [28]. Also, vMF embedding spaces have been studied for deep metric learning [9] since such hypersphere embedding spaces are more desirable than conventional Euclidean spaces when their dimension is large.…”