Natural gradient enables fast sampling in spiking neural networks

Abstract

For animals to navigate an uncertain world, their brains need to estimate uncertainty at the timescales of sensations and actions. Sampling-based algorithms afford a theoretically-grounded framework for probabilistic inference in neural circuits, but it remains unknown how one can implement fast sampling algorithms in biologically-plausible spiking networks. Here, we propose to leverage the population geometry, controlled by the neural code and the neural dynamics, to implement fast samplers in spiking neural networks. We first show that that two classes of spiking samplers—efficient balanced spiking networks that simulate Langevin sampling, and networks with probabilistic spike rules that implement Metropolis-Hastings sampling—can be unified within a common framework. We then show that careful choice of population geometry enables rapid inference of parameters drawn from strongly-correlated high-dimensional distributions in both networks. Our results suggest design principles for algorithms for sampling-based probabilistic inference in spiking neural networks, yielding potential inspiration for neuromorphic computing and testable predictions for neurobiology.

Neural Circuits for Fast Poisson Compressed Sensing in the Olfactory Bulb

“…Second, our models are rate-based, while neurons in the OB spike. In spiking implementations of sampling networks, the noise is not uncorrelated across neurons, complicating their biological interpretation [38,114].…”

“…Constructing models that capture these richly nonlinear effects will be an important objective for future work. A first step towards such a nonlinear model would be to build a spiking network that approximates the rate-based models considered here, which could be accomplished using the efficient balanced network formalism for distributed spiking networks [38]. Another step would be to add a Hill function nonlinearity to the OSN model to approximate competitive binding, as studied for Gaussian compressed sensing by Qin et al [26].…”

“…Here, the information geometry of the problem is governed by the sensing matrix A, which introduces correlations in the input signals to mitral cells because the off-diagonal components of A ⊤ A are non-negligible. To counteract these detrimental correlations, we can choose a readout matrix Γ such that ΓΓ ⊤ ≈ (A ⊤ A) + up to constants of proportionality, where we must take a pseudoinverse because A ⊤ A is highly rankdeficient [37][38][39][40] (see Appendix G for details). As a control, we also consider a naïvely distributed code with ΓΓ ⊤ ≈ I n odor .…”

“…Thus far, we have focused on a circuit that performs MAP estimation of odorant concentrations. However, to successfully navigate a dynamic, noisy world, animals must estimate sensory uncertainty at the timescale of perception [38,[103][104][105][106][107]. Fortunately, our circuit model can be easily extended to perform Langevin sampling of the full posterior distribution, allowing for uncertainty estimation while maintaining its attractive structural features.…”

“…Indeed, many proposals assume that the presence of each odorant is represented by a single, specialized neuron, which is inconsistent with the distributed odor coding observed in vivo [22,[34][35][36]. This axis-aligned coding does not leverage the geometric structure of the sensory space, which can be defined even in the absence of interpretable dimensions [37][38][39][40][41].…”