‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

Abstract

Distributions of data or sensory stimuli often enjoy underlying invariances. How and to what extent those symmetries are captured by unsupervised learning methods is a relevant question in machine learning and in computational neuroscience. We study here, through a combination of numerical and analytical tools, the learning dynamics of Restricted Boltzmann Machines (RBM), a neural network paradigm for representation learning. As learning proceeds from a random configuration of the network weights, we show the existence of, and characterize a symmetry-breaking phenomenon, in which the latent variables acquire receptive fields focusing on limited parts of the invariant manifold supporting the data. The symmetry is restored at large learning times through the diffusion of the receptive field over the invariant manifold; hence, the RBM effectively spans a continuous attractor in the space of network weights. This symmetry-breaking phenomenon takes place only if the amount of data available for training exceeds some critical value, depending on the network size and the intensity of symmetry-induced correlations in the data; below this 'retarded-learning' threshold, the network weights are essentially noisy and overfit the data. I. INTRODUCTIONMany high-dimensional inputs or data enjoy various kinds of low-dimensional invariances, which are at the basis of the socalled manifold hypothesis [1]. For instance, the pictures of somebody's face are related to each other through a set of continuous symmetries corresponding to the degrees of freedom characterizing the relative position of the camera (rotations, translations, changes of scales) as well as the internal deformations of the face (controlled by muscles). While well-understood symmetries can be explicitely taken care of through adequate procedures, e.g. convolutional networks, not all invariances may be known a priori. An interesting question is therefore if and how these residual symmetries affect the representations of the data achieved by learning models.This question does not arise solely in the context of machine learning, but is also of interest in computational neuroscience, where it is of crucial importance to understand how the statistical structure of input stimuli, be they visual, olfactive, auditory, tactile, ... shapes their encoding by sensory brain areas and their processing by higher cortical regions. Information theory provides a mathematical framework to answer this question [2], and was applied, in the case of linear models of neurons, to a variety of situations, including the prediction of the receptive fields of retinal ganglion cells [3], the determination of cone fractions in the human retina [4] or the efficient representation of odor-variable environments [5]. In the case of natural images, which enjoy approximate translational and rotational invariances, non-linear learning rules resulting from adequate modification of Oja's dynamics [6] or sparse-representation learning procedures [7] produce local edge detectors, such as do independent co...

Are place cells just memory cells? Memory compression leads to spatial tuning and history dependence

“…There is also another category of recent theoretical work that emphasizes the general role of the hippocampus in learning and memory (see e.g. (66)(67)(68)(69)(70)(71)(72)(73)) at the expense of the specific role it plays in navigation. These works mostly focus on the encoding of temporal sequences with one notable exception (73) and some assume that the hippocampus tries to store only the information that is relevant for predicting the next state of the environment, while others postulate that the goal of the hippocampus is to represent a whole probability distribution of future locations (conditioned on the current position of the animal).…”

Restricted Boltzmann machine: Recent advances and mean-field theory*

“…Again, the position of the peaks diffuse with time even though some repulsive interaction seems to forbid them to cross. See Figure 7 taken from [68] illustrating this phenomena. Right: figure from Ref.…”

“…The diffusion aims at reproducing the correlation patterns of the translational symmetry, while the repelling interaction ensure that two peaks will not overlap.. Now in Ref. [68], the author compute the gradient of a system with one hidden node…”

“…This equation can be cast into a spatial diffusion equation with additional term in the continuous time limit (see Ref. [68] for more details). From this small coupling expansion it is also possible to study the stationary solution in the one hidden node case and show that it is consistent with experimental results: it describes a peaked function decreasing rapidly as the distance from the center increases.…”

“…Each peak intends to reproduce the correlation pattern around a central node, and therefore cannot reproduce the translational symmetry of the problem. Right: figure from Ref [68],. the position of the three peaks as a function of the number of training epochs.…”