A Theoretical Framework for Inference Learning

Abstract

Backpropagation (BP) is the most successful and widely used algorithm in deep learning. However, the computations required by BP are challenging to reconcile with known neurobiology. This difficulty has stimulated interest in more biologically plausible alternatives to BP. One such algorithm is the inference learning algorithm (IL). IL has close connections to neurobiological models of cortical function and has achieved equal performance to BP on supervised learning and auto-associative tasks. In contrast to BP, however, the mathematical foundations of IL are not well-understood. Here, we develop a novel theoretical framework for IL. Our main result is that IL closely approximates an optimization method known as implicit stochastic gradient descent (implicit SGD), which is distinct from the explicit SGD implemented by BP. Our results further show how the standard implementation of IL can be altered to better approximate implicit SGD. Our novel implementation considerably improves the stability of IL across learning rates, which is consistent with our theory, as a key property of implicit SGD is its stability. We provide extensive simulation results that further support our theoretical interpretations and also demonstrate IL achieves quicker convergence when trained with small mini-batches while matching the performance of BP for large mini-batches.

Gibbs sampling the posterior of neural networks

“…We compare these methods in the case of a one hidden layer neural network, identical for the teacher and the student, with input dimension d 1 = 50, d 2 = 10 hidden units and a scalar output. This corresponds to the function (2) σ W (1) x + b (1) , (11) where σ(x) = max(0, x) and W indicates the collection of all parameters: W (1) ∈ R d2×d1 and W (2) ∈ R 1×d2 . We specify the prior by setting λ…”

“…, where P indicates the posterior distribution; this is the score rescaled by ∆ and averaged over the first layer weights. We also tried taking the gradient with respect to the second layer weights W (2) , or with respect to Z (2) . The results do not change significantly and the score methods keep severely underestimating the thermalization time.…”

“…As an alternative, other works have introduced the 'stochastic feedforward networks', where noise is added at every layer's pre-activation [35,40,44,54]. Outside of the Bayesian learning of neural networks literature, models where intermediate pre-or postactivations are added as dynamical variables have also been considered in the predictive coding literature [2,33,34,51].…”

“…In the notation of A, θ is given by {W (1) , b (1) , Z (2) , X (2) , W (2) , b (2) }. Next we have to pick the (possibly vector-valued) function ψ(θ).…”

“…Notice that operations within each of the inner loops over ℓ can be executed in parallel, as there are no cross dependencies between different ℓs: for example, one can sample X (2) , X (3) and X (4) in parallel. This implies that, provided that we have enough computing power, the running time of the algorithm can be made independent of the depth of the network.…”