A Bayesian Interpretation of the Light Gated Recurrent Unit

Abstract

We summarise previous work showing that the basic sigmoid activation function arises as an instance of Bayes's theorem, and that recurrence follows from the prior. We derive a layerwise recurrence without the assumptions of previous work, and show that it leads to a standard recurrence with modest modifications to reflect use of log-probabilities. The resulting architecture closely resembles the Li-GRU which is the current state of the art for ASR. Although the contribution is mainly theoretical, we show that it is able to outperform the state of the art on the TIMIT and AMI datasets.

A surrogate gradient spiking baseline for speech command recognition

“…The gated recurrent unit (GRU) of Cho et al (2014) and the light GRU (liGRU) of Ravanelli et al (2018) constitute gradual simplifications of the LSTM with fewer gates in an effort to reduce the size of recurrent units. Very recently, the authors have derived a probabilistically interpretable version of the liGRU called light Bayesian recurrent unit (liBRU) that showed slight improvements over the liGRU on speech recognition tasks (Bittar and Garner, 2021). We will implement MLPs, RNNs, liBRUs, and GRUs, which will serve as an ANNbaseline to compare with our SNNs.…”

Bayesian Recurrent Units and the Forward-Backward Algorithm

“…Similarly to [19], for the case of multivariate normal distributions that share the same covariance matrix Σ, i.e., p(xt|φt) ∼ N (µ, Σ) and p(xt| ¬φt ) ∼ N (ν, Σ), the parameters W ∈ R F ×H and b ∈ R H can be expressed as,…”

“…Recurrence emerges naturally from Bayes's theorem which updates a prior probability into a posterior given new observational data. In our previous work on the light Bayesian recurrent unit [19], hidden features were assumed to be interdependent, which led to a layer-wise recurrence for the computation of prior probabilities. In this paper, in a mainly theoretical contribution, we come back to the simpler case of a RNN with unit-wise recurrence and no gate.…”