Manifold-tiling Localized Receptive Fields are Optimal in Similarity-preserving Neural Networks

Sengupta, Anirvan M.; Tepper, Mariano; Pehlevan, Cengiz; Genkin, Alexander; Chklovskii, Dmitri B.

doi:10.1101/338947

Cited by 26 publications

(58 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further supporting evidence was recently brought by several works, focusing on the production of such receptive fields in the context of unsupervised learning. Learning of symmetric data with similarity-preserving representations [10] or with auto-encoders [11] both led to localized receptive fields tiling the underlying manifold, in striking analogy with place cells and spatial maps in the hippocampus. In turn, such high-dimensional place-cell-like representations have putative functional advantages: they can be efficiently and accurately learned by recurrent neural networks, and thus allow for the storage and retrieval of multiple cognitive low-dimensional maps [12].The present work is an additional effort to investigate this issue in a highly simplified and idealized framework of unsupervised learning, where both the data distribution and the machine are under full control.…”

mentioning

confidence: 99%

‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

Harsh¹,

Tubiana²,

Cocco³

et al. 2020

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

Distributions of data or sensory stimuli often enjoy underlying invariances. How and to what extent those symmetries are captured by unsupervised learning methods is a relevant question in machine learning and in computational neuroscience. We study here, through a combination of numerical and analytical tools, the learning dynamics of Restricted Boltzmann Machines (RBM), a neural network paradigm for representation learning. As learning proceeds from a random configuration of the network weights, we show the existence of, and characterize a symmetry-breaking phenomenon, in which the latent variables acquire receptive fields focusing on limited parts of the invariant manifold supporting the data. The symmetry is restored at large learning times through the diffusion of the receptive field over the invariant manifold; hence, the RBM effectively spans a continuous attractor in the space of network weights. This symmetry-breaking phenomenon takes place only if the amount of data available for training exceeds some critical value, depending on the network size and the intensity of symmetry-induced correlations in the data; below this 'retarded-learning' threshold, the network weights are essentially noisy and overfit the data. I. INTRODUCTIONMany high-dimensional inputs or data enjoy various kinds of low-dimensional invariances, which are at the basis of the socalled manifold hypothesis [1]. For instance, the pictures of somebody's face are related to each other through a set of continuous symmetries corresponding to the degrees of freedom characterizing the relative position of the camera (rotations, translations, changes of scales) as well as the internal deformations of the face (controlled by muscles). While well-understood symmetries can be explicitely taken care of through adequate procedures, e.g. convolutional networks, not all invariances may be known a priori. An interesting question is therefore if and how these residual symmetries affect the representations of the data achieved by learning models.This question does not arise solely in the context of machine learning, but is also of interest in computational neuroscience, where it is of crucial importance to understand how the statistical structure of input stimuli, be they visual, olfactive, auditory, tactile, ... shapes their encoding by sensory brain areas and their processing by higher cortical regions. Information theory provides a mathematical framework to answer this question [2], and was applied, in the case of linear models of neurons, to a variety of situations, including the prediction of the receptive fields of retinal ganglion cells [3], the determination of cone fractions in the human retina [4] or the efficient representation of odor-variable environments [5]. In the case of natural images, which enjoy approximate translational and rotational invariances, non-linear learning rules resulting from adequate modification of Oja's dynamics [6] or sparse-representation learning procedures [7] produce local edge detectors, such as do independent co...

show abstract

mentioning

confidence: 99%

‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

Harsh¹,

Tubiana²,

Cocco³

et al. 2020

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

show abstract

“…To solve the objective (17) in the online setting, we introduce the constraints in the cost via Lagrange multipliers and using the variable substitution trick, we can derive a NN implementation of this algorithm [31] (Fig. 4A).…”

Section: A Similarity-based Cost Function and Nn For Clusteringmentioning

confidence: 99%

Neuroscience-Inspired Online Unsupervised Learning Algorithms: Artificial Neural Networks

Pehlevan

Chklovskii

2019

IEEE Signal Process. Mag.

Self Cite

View full text Add to dashboard Cite

Although the currently popular deep learning networks achieve unprecedented performance on some tasks, the human brain still has a monopoly on general intelligence. Motivated by this and biological implausibility of deep learning networks, we developed a family of biologically plausible artificial neural networks (NNs) for unsupervised learning. Our approach is based on optimizing principled objective functions containing a term that matches the pairwise similarity of outputs to the similarity of inputs, hence the name -similarity-based. Gradient-based online optimization of such similarity-based objective functions can be implemented by NNs with biologically plausible local learning rules. Similarity-based cost functions and associated NNs solve unsupervised learning tasks such as linear dimensionality reduction, sparse and/or nonnegative feature extraction, blind nonnegative source separation, clustering and manifold learning.

show abstract

“…, T , m being the number of output channels, or hidden units in our two-layer network, Figure 1, left. Manifold-tiling networks have been derived [8] from similarity-preserving objectives [5] with a non-negativity constraint. Similarity preservation postulates that similar input pairs, x t and x t , evoke similar output pairs, h t and h t .…”

Section: Review Of the Manifold-tiling Network Derived Frommentioning

confidence: 99%

“…with the same update for w as in Eq. (8). The behavior of both algorithms is almost indistinguishable, so we only report the results from Eq.…”

Section: A Neural Network For Semi-supervised Learningmentioning

confidence: 99%

“…The manifold structure is captured by the correlations between the components carried by corresponding channels. Because most existing algorithms for sparse representations, such as [10], do not have natural neural implementations we base our work on the recently developed manifold tiling algorithm [8]. Inspiration for such design comes from biological neural networks such as place cells in the rodent hippocampus.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Neural Network for Semi-supervised Learning on Manifolds

Genkin¹,

Sengupta²,

Chklovskii³

2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Semi-supervised learning algorithms typically construct a weighted graph of data points to represent a manifold. However, an explicit graph representation is problematic for neural networks operating in the online setting. Here, we propose a feed-forward neural network capable of semi-supervised learning on manifolds without using an explicit graph representation. Our algorithm uses channels that represent localities on the manifold such that correlations between channels represent manifold structure. The proposed neural network has two layers. The first layer learns to build a representation of low-dimensional manifolds in the input data as proposed recently in [8]. The second learns to classify data using both occasional supervision and similarity of the manifold representation of the data. The channel carrying label information for the second layer is assumed to be "silent" most of the time. Learning in both layers is Hebbian, making our network design biologically plausible. We experimentally demonstrate the effect of semi-supervised learning on non-trivial manifolds.

show abstract

Manifold-tiling Localized Receptive Fields are Optimal in Similarity-preserving Neural Networks

Cited by 26 publications

References 29 publications

‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

Neuroscience-Inspired Online Unsupervised Learning Algorithms: Artificial Neural Networks

A Neural Network for Semi-supervised Learning on Manifolds

Contact Info

Product

Resources

About