CSNNs: Unsupervised, Backpropagation-Free Convolutional Neural Networks for Representation Learning

Abstract

This work combines Convolutional Neural Networks (CNNs), clustering via Self-Organizing Maps (SOMs) and Hebbian Learning to propose the building blocks of Convolutional Self-Organizing Neural Networks (CSNNs), which learn representations in an unsupervised and Backpropagation-free manner. Our approach replaces the learning of traditional convolutional layers from CNNs with the competitive learning procedure of SOMs and simultaneously learns local masks between those layers with separate Hebbian-like learning rules to overcome the problem of disentangling factors of variation when filters are learned through clustering. We investigate the learned representation by designing two simple models with our building blocks, achieving comparable performance to many methods which use Backpropagation, while we reach comparable performance on Cifar10 and give baseline performances on Cifar100, Tiny ImageNet and a small subset of ImageNet for Backpropagationfree methods.

Deep Learning Self-Organizing Map of Convolutional Layers

“…If these are provided they can potentially be used, but, typically, an unsupervised model should be is in position to function even when these are absent or missing. Nevertheless, there is merely a handful of approaches that adhere to strict unsupervised training criteria [5], [6], [7]and the ones reporting results on the MNIST database are specifically indicated in Table 1. Frequently, in the literature, an "unsupervised" model with a supervised or self/semi-supervised training procedure is proposed.…”

“…building a deep SOM and training it in a purely unsupervised way has proven to be a complex and difficult task. Only a small number of models exist that can be classified as unsupervised beyond any doubt [5], [6]and [7]. Equally few are the approaches that extend beyond the three hidden layer limit [8], [9] and [6].…”

Deep Self-Organizing Map of Convolutional Layers for Clustering and Visualizing Image Data

“…They have been well studied and their early-stage testing value is undoubtable, but when used in isolation they might offer a partial biased view of a model's capabilities. For instance, their grayscale characteristic (i.e., that they consist of single-channel images) conceals the fact that a substantial number of deep SOMs are not in a position to process and model colored images whereas a handful of advanced ones like [14][15][16]47] succeed in doing. Nevertheless, preliminary results for the MNIST benchmark of a pilot SOCOM study can be found in [48].…”

“…Meeting both main objectives i.e., building a deep SOM and training it in a purely unsupervised way has proven a complex and difficult task. Only a small number of models exist that can be classified as unsupervised beyond any doubt [12][13][14]. Equally few are the approaches that extend beyond the three hidden layer limit [13,15,16].…”

Solving the Feature Diversity Problem Based on Multi-Model Scheme

“…By using fusion method, the classification accuracy may be increased. We evaluated the methods on some real datasets that include CIFAR-10 [11-13] and CIFAR-100 [14][15][16]. Furthermore, we also collected a real dataset where the samples have different resolution and each label has different number of samples.…”