Generalization Learning in a Perceptron with Binary Synapses

Abstract

We consider the generalization problem for a perceptron with binary synapses, implementing the Stochastic BeliefPropagation-Inspired (SBPI) learning algorithm which we proposed earlier, and perform a mean-field calculation to obtain a differential equation which describes the behaviour of the device in the limit of a large number of synapses N . We show that the solving time of SBPI is of order N √ log N , while the similar, well-known clipped perceptron (CP) algorithm does not converge to a solution at all in the time frame we considered. The analysis gives some insight into the ongoing process and shows that, in this context, the SBPI algorithm is equivalent to a new, simpler algorithm, which only differs from the CP algorithm by the addition of a stochastic, unsupervised meta-plastic reinforcement process, whose rate of application must be less than 2/ (πN ) for the learning to be achieved effectively. The analytical results are confirmed by simulations.

Binary Synapse Circuitry for High Efficiency Learning Algorithm Using Generalized Boundary Condition Memristor Models

“…Here, we preliminary describe a simplified version of that algorithm (called CP in [6]), and we present a circuit which implements it. The CP algorithm has a reduced performance with respect to the SBPI algorithm, but our scheme can be extended rather straightforwardly to SBPI, or a variant thereof (such as the one proposed in [7]), since it is already able to model the most crucial quantities used in the algorithm; the extension to complete SBPI algorithm will be the subject of a future work, currently in preparation. A set ξ of binary patterns is presented to a network of N .…”

Active Online Learning in the Binary Perceptron Problem

“…Theoretical and numerical studies on various algorithms have found that the generalization error ε of the passive learning process decreases with the pattern density α algebraically, e.g., ε ∝ α −1 . This means that in the thermodynamic limit of N → ∞, perfect inference is unlikely to achieve at any finite value of pattern density α [13][14][15][16][17][18][19].…”

Role of Synaptic Stochasticity in Training Low-Precision Neural Networks

“…In nature, synaptic weights are known to be plastic, low precision and unreliable, and it is an interesting issue to understand if this stochasticity can help learning or if it is an obstacle. The debate about this issue has a long history and is still unresolved (see [1] and references therein). Here, we provide quantitative evidence that the stochasticity associated with noisy low precision synapses can drive elementary supervised learning processes towards a particular type of solutions which, despite being rare, are robust to noise and generalize well -two crucial features for learning processes.In recent years, multi-layer (deep) neural networks have gained prominence as powerful tools for tackling a large number of cognitive tasks [2].…”

“…Also notice that for any maximizer W of Problem (1) we have that δ (W − W ) is a maximizer of Problem (3) provided that it belongs to the parametric family, as can be shown using Jensen's inequality. Problem (3) is a "distributional" relaxation of Problem (1).…”

“…The partition function depends implicitly on the inverse temperature β, a training set D = {y µ , x µ } M µ=1 , M = αN for some α > 0, y µ ∈ {−1, 1} , x µ ∈ {−1, 1} N , and a norm parameter q . The integration is in the box Ω = [− 1,1] N . The log-likelihood is given, for large N , by…”