Learning sparse codes from compressed representations with biologically plausible local wiring constraints

Abstract

Sparse coding is an important method for unsupervised learning of task-independent features in theoretical neuroscience models of neural coding. While a number of algorithms exist to learn these representations from the statistics of a dataset, they largely ignore the information bottlenecks present in fiber pathways connecting cortical areas. For example, the visual pathway has many fewer neurons transmitting visual information to cortex than the number of photoreceptors. Both empirical and analytic results have recently shown that sparse representations can be learned effectively after performing dimensionality reduction with randomized linear op- erators, producing latent coefficients that preserve information. Unfortunately, current proposals for sparse coding in the compressed space require a centralized compression process (i.e., dense random matrix) that is biologically unrealistic due to local wiring constraints observed in neural circuits. The main contribution of this paper is to leverage recent results on structured random matrices to propose a theoretical neuroscience model of randomized projections for communication be- tween cortical areas that is consistent with the local wiring constraints observed in neuroanatomy. We show analytically and empirically that unsupervised learning of sparse representations can be performed in the compressed space despite significant local wiring constraints in compression matrices of varying forms (corresponding to different local wiring patterns). Our analysis verifies that even with significant local wiring constraints, the learned representations remain qualitatively similar, have similar quantitative performance in both training and generalization error, and are consistent across many measures with measured macaque V1 receptive fields.

Structured random receptive fields enable informative sensory encodings

“…Random compression matrices are known to have optimal properties, however in many cases structured randomness is more realistic. Recent work has shown that structured random projections with local wiring constraints (in one dimension) was compatible with dictionary learning [25], supporting previous empirical results [5]. Our work shows that structured random receptive fields are equivalent to employing a wavelet dictionary and dense Gaussian projection.…”

“…We see similar results across diverse hidden layer widths and learning rates (Fig. [25][26][27][28], with the benefits most evident for wider networks and smaller learning rates. Furthermore, the structured weights show similar results when trained for 10,000 epochs (rate 0.1; 1,000 neurons; not shown) and with other optimizers like minibatch Stochastic Gradient Descent (SGD) and ADAM (batch size 256, rate 0.1; 1,000 neurons; not shown).…”

“…To fit the covariance model to the data, we optimize the parameters (see Appendix A.3) f lo , f hi , and γ, finding f lo = 75 Hz, f hi = 200 Hz, and γ = 12.17 ms best fit the sensilla data. We do so by minimizing the Frobenius norm of the difference between C data and the model (26). The resulting model covariance matrix (Fig.…”

“…We show results of initializing fully trained neural networks across a range of network widths (50, 100, 400, and 1,000) and learning rates (10 −3 , 10 −2 , and 10 −1 ) in Figures 25,26…”

Variational Sparse Coding with Learned Thresholding

“…, of the inferred codes [15,13,7]. This quantity is minimized when the features have high joint entropy while being independent of one another, denoting a reduction in redundancy.…”