Sparse Coding Enables the Reconstruction of High-Fidelity Images and Video from Retinal Spike Trains

Abstract

The optic nerve transmits visual information to the brain as trains of discrete events, a low-power, low-bandwidth communication channel also exploited by silicon retina cameras. Extracting highfidelity visual input from retinal event trains is thus a key challenge for both computational neuroscience and neuromorphic engineering. Here, we investigate whether sparse coding can enable the reconstruction of high-fidelity images and video from retinal event trains. Our approach is analogous to compressive sensing, in which only a random subset of pixels are transmitted and the missing information is estimated via inference. We employed a variant of the Locally Competitive Algorithm to infer sparse representations from retinal event trains, using a dictionary of convolutional features optimized via stochastic gradient descent and trained in an unsupervised manner using a local Hebbian learning rule with momentum. We used an anatomically realistic retinal model with stochastic graded release from cones and bipolar cells to encode thumbnail images as spike trains arising from ON and OFF retinal ganglion cells. The spikes from each model ganglion cell were summed over a 32 msec time window, yielding a noisy rate-coded image. Analogous to how the primary visual cortex is postulated to infer features from noisy spike trains arising from the optic nerve, we inferred a higher-fidelity sparse reconstruction from the noisy rate-coded image using a convolutional dictionary trained on the original CIFAR10 database. To investigate whether a similar approach works on non-stochastic data, we demonstrate that the same procedure can be used to reconstruct high-frequency video from the asynchronous events arising from a silicon retina camera moving through a laboratory environment.

Modeling Biological Immunity to Adversarial Examples

“…photoreceptors, bipolar, horizontal, amacrine, and ganglion cells. Our architecture extends a previously described retinal model that sought to explain synchronous, stimulus-selective oscillations between retinal ganglion cells [25,46]. Figure 4: Retina Ganglion Cell ON (red) and OFF (blue) action potentials from ISLVRC2012 val 1003, 1-28 ms. Each image frame displays a 1 ms snapshot.…”

Deep Event Stereo Leveraged by Event-to-Image Translation

“…In this study, we propose a novel end-to-end deep event stereo architecture to generate spatial image features from input event data and use them as a guidance for the accurate stereo event matching. Inspired by the recent studies (Kalia, Navab, and Salcudean 2019;Rebecq et al 2019;Scheerlinck The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) VidalMata et al 2019;Wang et al 2019;Watkins et al 2018), we explicitly reconstruct intensity images from the input event streams and use them as a guidance for event features. By doing so, we can use not only asynchronous event information but also spatial intensity image information.…”

Unsupervised Dictionary Learning via a Spiking Locally Competitive Algorithm

“…The Locally Competitive Algorithm (LCA) describes a dynamical neural network that uses only local synaptic interactions between non-spiking leaky-integrator neurons to infer sparse representations of input stimuli [10]. Unsupervised dictionary learning using convolutional LCA [12] has been used to infer sparse representations that support a number of signal processing tasks [17] [16][8] [7]. However, as previously implemented, unsupervised learning with non-spiking LCA utilizes non-local computations, specifically transpose and normalization operations performed globally on the entire weight matrix, and further requires signed outputs in order to represent the sparse reconstruction error.…”