Sparse and silent coding in neural circuits

Abstract

Sparse coding algorithms are about finding a linear basis in which signals can be represented by a small number of active (non-zero) coefficients. Such coding has many applications in science and engineering and is believed to play an important role in neural information processing. However, due to the computational complexity of the task, only approximate solutions provide the required efficiency (in terms of time). As new results show, under particular conditions there exist efficient solutions by minimizing the magnitude of the coefficients ('l 1 -norm') instead of minimizing the size of the active subset of features ('l 0 -norm'). Straightforward neural implementation of these solutions is not likely, as they require a priori knowledge of the number of active features. Furthermore, these methods utilize iterative re-evaluation of the reconstruction error, which in turn implies that final sparse forms (featuring 'population sparseness') can only be reached through the formation of a series of non-sparse representations, which is in contrast with the overall sparse functioning of the neural systems ('lifetime sparseness'). In this article we present a novel algorithm which integrates our previous 'l 0 -norm' model on spike based probabilistic optimization for sparse coding with ideas coming from novel 'l 1 -norm' solutions.The resulting algorithm allows neurally plausible implementation and does not require an exactly defined sparseness level thus it is suitable for representing natural stimuli with a varying number of features. We also demonstrate that the combined method significantly extends the domain where optimal solutions can be found by 'l 1 -norm' based algorithms.

Cell recognition based on topological sparse coding for microscopy imaging of focused ultrasound treatment

“…These are commonly-used quantitative metrics to evaluate the performance of mitotic cell recognition. λ and γ are again trade-off parameters controlling the balance between the reconstruction quality and sparsity [24, 25], when comparing the performance of different dictionary learning strategies with four visual features and different configurations, λ and γ were set to 0.1 [26, 27] and C is set to 1.…”

Efficient Sparse Coding in Early Sensory Processing: Lessons from Signal Recovery

“…However, overcompleteness presents a non-trivial challenge for computational models on neural representations. In comparison with biological data, most computational models of SC can find the optimal solution if overcompleteness is 2 to 8-fold at most [16] . Importantly, higher level of overcompleteness may increase the overall metabolic cost of neural coding for two reasons.…”

“…Its main limitation is the slow convergence rate. The combination, termed Subspace Cross-Entropy (SCE) [16] method inherits the best of both worlds: it is reasonably fast and still can yield the optimal solution even at a higher level of overcompleteness. Since we are interested in the formation of sparse codes at very high level of overcompleteness, we used SCE in our numerical experiments.…”

“…The appendix contains the pseudocodes of SP, CEM and SCE for the sake of reproducibility. Detailed analysis of these methods can be found in [16] , [28] , [29] .…”

“…Subspace Cross-Entropy method (SCE) is an efficient combination of CEM and SP for overcomplete sparse coding. A detailed description can be found in [16] and the pseudocode is given in Table 3 . SCE inherits the flexibility and synaptic efficiency of CEM as well as the superior speed and scaling properties of SP without their shortcomings.…”