Evaluating Hebbian Learning in a Semi-supervised Setting

Abstract

We propose a semi-supervised learning strategy for deep Convolutional Neural Networks (CNNs) in which an unsupervised pretraining stage, performed using biologically inspired Hebbian learning algorithms, is followed by supervised end-to-end backprop fine-tuning. We explored two Hebbian learning rules for the unsupervised pre-training stage: soft-Winner-Takes-All (soft-WTA) and nonlinear Hebbian Principal Component Analysis (HPCA). Our approach was applied in sample efficiency scenarios, where the amount of available labeled training samples is very limited, and unsupervised pre-training is therefore beneficial. We performed experiments on CIFAR10, CIFAR100, and Tiny ImageNet datasets. Our results show that Hebbian outperforms Variational Auto-Encoder (VAE) pre-training in almost all the cases, with HPCA generally performing better than soft-WTA.

Deep Features for CBIR with Scarce Data using Hebbian Learning

“…The details of Hebbian learning theories are outside the scope of this paper, although a plethora of Hebbian-based learning algorithms have been proposed. We refer the interested reader to [1,13,[24][25][26][27]. In particular, in our previous works [24,25], we have highlighted the merits of the nonlinear Hebbian Principal Component Analysis (HPCA).…”

“…We refer the interested reader to [1,13,[24][25][26][27]. In particular, in our previous works [24,25], we have highlighted the merits of the nonlinear Hebbian Principal Component Analysis (HPCA). This is derived by minimizing the representation error:…”

“…Where 𝜂 is the learning rate. Since we found this learning rule to be particularly effective in semi-supervised regimes with label scarcity [24,25], this is the rule that we adopted in the unsupervised pre-training phase of our method.…”

“…Unsupervised pre-training is known to be beneficial before further fine-tuning stage [7,19,29,38]. Concerning the Hebbian pre-training phase, a nonlinear learning rule for Hebbian Principal Component Analysis (HPCA) was considered [24,25,27]. For the supervised fine-tuning stage, we considered sample efficiency scenarios, in which only a fraction of the training data is assumed to be labeled.…”

Hebbian semi-supervised learning in a sample efficiency setting

“…Therefore, researchers took again inspiration from biology, in order to find new learning solutions as alternatives to backprop. The goal was not only to address the problem of SNN training [33,148], but also to discover novel approaches to the learning problem [77,139,182], and possibly more data efficient strategies [69,90,[105][106][107]110]. In fact, another limitation of current DL solutions is the requirement of Fig.…”