Soft-bound Synaptic Plasticity Increases Storage Capacity

Abstract

Accurate models of synaptic plasticity are essential to understand the adaptive properties of the nervous system and for realistic models of learning and memory. Experiments have shown that synaptic plasticity depends not only on pre- and post-synaptic activity patterns, but also on the strength of the connection itself. Namely, weaker synapses are more easily strengthened than already strong ones. This so called soft-bound plasticity automatically constrains the synaptic strengths. It is known that this has important consequences for the dynamics of plasticity and the synaptic weight distribution, but its impact on information storage is unknown. In this modeling study we introduce an information theoretic framework to analyse memory storage in an online learning setting. We show that soft-bound plasticity increases a variety of performance criteria by about 18% over hard-bound plasticity, and likely maximizes the storage capacity of synapses.

Energy Efficient Sparse Connectivity from Imbalanced Synaptic Plasticity Rules

“…So far we assumed the inputs were -1 or +1, as in earlier studies of the non-negative perceptron [ 9 , 26 , 27 ]. This is hard to imagine biologically, unless an inhibitory partner neuron is introduced [ 19 , 31 , 41 , 42 ]. An arguably more faithful biological model is obtained by representing low inputs as silent, x i = 0 [ 16 , 19 , 20 , 28 , 43 ].…”

“…This is hard to imagine biologically, unless an inhibitory partner neuron is introduced [ 19 , 31 , 41 , 42 ]. An arguably more faithful biological model is obtained by representing low inputs as silent, x i = 0 [ 16 , 19 , 20 , 28 , 43 ]. Furthermore, we wish to generalise to a case where the probability for a high input is variable rather than fixed to 1/2.…”

“…This can be viewed as a kind of homeostatic adaptation process: as learning progresses, the neuron self-regulates so that it becomes harder to reach the firing threshold. While the incorporation of an auxiliary feed-forward inhibition circuit has been used in related models to increase capacity in the presence of non-negativity constraints [ 19 , 31 , 41 , 42 ], the mechanism here does not directly depend on the precise pattern x of the presented input. It thereby obviates the need for coordinated plasticity with a partner interneuron as well as for precise temporal integration of inhibitory signals.…”

“…We imbalance potentiation and depression to achieve sparse connectivity. In other memory tasks, the information loss can be substantial for imbalanced plasticity; for instance, postsynaptic-independent (i.e., without a stop-learning mechanism) online learning rules are severely affected when depression does not match potentiation [ 17 – 19 ]. However, here imbalance leads to a substantial energy reduction in storage as long as the task is below maximal capacity.…”

“…Here we explore imbalanced plasticity as a simple and biologically plausible way to reduce the number of required synapses and thus improve information storage efficiency. In many memory models the amount of potentiation and depression are precisely matched to the statistics of the neural activity [ 16 – 19 ], but here we deliberately perturb the optimal plasticity rule by introducing a bias towards depression. This imbalanced plasticity finds weight configurations that require less functional synapses and that are thus more energy efficient.…”