Structural Plasticity Denoises Responses and Improves Learning Speed

Abstract

Despite an abundance of computational models for learning of synaptic weights, there has been relatively little research on structural plasticity, i.e., the creation and elimination of synapses. Especially, it is not clear how structural plasticity works in concert with spike-timing-dependent plasticity (STDP) and what advantages their combination offers. Here we present a fairly large-scale functional model that uses leaky integrate-and-fire neurons, STDP, homeostasis, recurrent connections, and structural plasticity to learn the input encoding, the relation between inputs, and to infer missing inputs. Using this model, we compare the error and the amount of noise in the network's responses with and without structural plasticity and the influence of structural plasticity on the learning speed of the network. Using structural plasticity during learning shows good results for learning the representation of input values, i.e., structural plasticity strongly reduces the noise of the response by preventing spikes with a high error. For inferring missing inputs we see similar results, with responses having less noise if the network was trained using structural plasticity. Additionally, using structural plasticity with pruning significantly decreased the time to learn weights suitable for inference. Presumably, this is due to the clearer signal containing less spikes that misrepresent the desired value. Therefore, this work shows that structural plasticity is not only able to improve upon the performance using STDP without structural plasticity but also speeds up learning. Additionally, it addresses the practical problem of limited resources for connectivity that is not only apparent in the mammalian neocortex but also in computer hardware or neuromorphic (brain-inspired) hardware by efficiently pruning synapses without losing performance.

A neuromimetic approach to the serial acquisition, long-term storage, and selective utilization of overlapping memory engrams

“…The other design tension we encounter arises between learning flexibility and the desirability of pruning candidate synapses for the sake of noise reduction. The noise reduction theme arises quite consistently across computational studies of structural plasticity: we found that candidate synapses are a noise source ( Table 2, rows 10 and 11), Spiess et al (30) found that well-pruned networks are less noisy and therefore learn faster than their unpruned counterparts, and Knoblaugh et al (29) found that networks have greater information capacity per synapse when some synapses can be pruned altogether rather than simply weakened. In our system, the allowance of a limited population of weak exploratory candidates is sufficient to allow for learning flexibility, but not enough to enable rapid learning.…”

“…The major outcome of our comparisons of the LASG, high-candidate, and lowcandidate regimes is that candidate synapses in general and their LASG-driven generation in particular are detrimental to response specificity. This is the result that we would expect, upon considering biologically-observed pruning and previous theoretical work that demonstrates pruning's noise-reduction (29,30) benefits. Specifically, in PING's presence, for all four input orderings, applied to both ISNs, response specificity is higher for both no-LASG conditions than for the LASG condition ( Table 2, rows 3 and 4).…”

Self-Evolutionary Neuron Model for Fast-Response Spiking Neural Networks

“…In [20], Zambrano et al presented an adaptive spiking neurons based network, where the neurons encode information in spike-trains using a form of Asynchronous Pulsed Sigma-Delta coding, and the authors demonstrated that the proposed neuron models based network responds an order of magnitude faster and uses an order of magnitude fewer spikes. In the structural plasticity mechanism which proposed in [21] that demonstrated this plasticity improves the learning speed of SNNs, the authors also took the neural conductance into account, and designed the conductance variation models for excitatory neurons and inhibitory neurons, respectively. In [21], the conductance variation models are defined as negative exponential relationships with time only, and the models are nonlinearly fused the spiking neuron model (leaky integrate-and-fire model, LIF model).…”

“…In the structural plasticity mechanism which proposed in [21] that demonstrated this plasticity improves the learning speed of SNNs, the authors also took the neural conductance into account, and designed the conductance variation models for excitatory neurons and inhibitory neurons, respectively. In [21], the conductance variation models are defined as negative exponential relationships with time only, and the models are nonlinearly fused the spiking neuron model (leaky integrate-and-fire model, LIF model).…”

“…In [20], the authors consider the firing threshold plasticity based adaptive spiking neuron model, but as the paper demonstrated, the presented networks are specific in straightforward neural networks without additional operations like pooling, softmax, etc., specific SNN variations need to be further developed. On the other hand, the structural plasticity proposed in [21] is to consider the network performance optimization problem from the aspect of connection creation or elimination between neurons, rather than the plasticity of some factors or variables of the neurons themselves.…”