Finding informative neurons in the brain using Multi-Scale Relevance

Self Cite

Deep learning has been successfully applied to various tasks, but its underlying mechanism remains unclear. Neural networks associate similar inputs in the visible layer to the same state of hidden variables in deep layers. The fraction of inputs that are associated to the same state is a natural measure of similarity and is simply related to the cost in bits required to represent these inputs. The degeneracy of states with the same information cost provides instead a natural measure of noise and is simply related the entropy of the frequency of states, that we call relevance. Representations with minimal noise, at a given level of similarity (resolution), are those that maximise the relevance. A signature of such efficient representations is that frequency distributions follow power laws. We show, in extensive numerical experiments, that deep neural networks extract a hierarchy of efficient representations from data, because they i) achieve low levels of noise (i.e. high relevance) and ii) exhibit power law distributions. We also find that the layer that is most efficient to reliably generate patterns of training data is the one for which relevance and resolution are traded at the same price, which implies that frequency distribution follows Zipf's law.

Section: Discussionmentioning

confidence: 99%

Resolution and relevance trade-offs in deep learning

Song

Marsili

2018

Self Cite

“…Eq. (20) then implies that H[E] increases by ν bits. The region ν > 0 corresponds to "redundant" representations, whereas for ν < 0, some informative bits are "lost in compression".…”

Section: The Thermodynamics Of Efficient Representationsmentioning

confidence: 99%

“…As such, we believe that our results may provide a guiding principle to extract relevant variables from high dimensional data (see e.g. [19,20]), or to shed light on the principles underlying deep learning (see e.g. [21,22]).…”

mentioning

confidence: 95%

Statistical criticality arises in most informative representations

Cubero

Marsili

et al. 2019

Self Cite

We show that statistical criticality, i.e. the occurrence of power law frequency distributions, arises in samples that are maximally informative about the underlying generating process. In order to reach this conclusion, we first identify the frequency with which different outcomes occur in a sample, as the variable carrying useful information on the generative process. The entropy of the frequency, that we call relevance, provides an upper bound to the number of informative bits. This differs from the entropy of the data, that we take as a measure of resolution. Samples that maximise relevance at a given resolution -that we call maximally informative samples -exhibit statistical criticality. In particular, Zipf's law arises at the optimal trade-off between resolution (i.e. compression) and relevance. As a byproduct, we derive a bound of the maximal number of parameters that can be estimated from a dataset, in the absence of prior knowledge on the generative model. Furthermore, we relate criticality to the statistical properties of the representation of the data generating process. We show that, as a consequence of the concentration property of the Asymptotic Equipartition Property, representations that are maximally informative about the data generating process are characterised by an exponential distribution of energy levels. This arises from a principle of minimal entropy, that is conjugate of the maximum entropy principle in statistical mechanics. This explains why statistical criticality requires no parameter fine tuning in maximally informative samples.When data are generated as independent draws from a parametric distribution, one can draw a sharp distinction between noise and useful information, that part of the data that can be used to estimate the generative model. Useful information is concentrated in sufficient statistics, which are those variables whose empirical value suffices to fully estimate the model's parameter [1]. The first aim of this paper is to draw the same distinction in the case where the model is not known. In this case, we show that the information on the 1 arXiv:1808.00249v5 [physics.data-an]

“…still follows the Gibbs distribution Eq. (19), but the value of β is dominated by the variables t if γ v < γ y and by the variables z otherwise 9 . Therefore, the most relevant set of variables are those with the smallest value of γ.…”

Section: Discussionmentioning

confidence: 99%

The peculiar statistical mechanics of optimal learning machines

Marsili¹

2019

Self Cite

Optimal Learning Machines (OLM) are systems that extract maximally informative representation of the environment they are in contact with, or of the data they are presented. It has recently been suggested that these systems are characterised by an exponential distribution of energy levels. In order to understand the peculiar properties of OLM within a broader framework, I consider an ensemble of optimisation problems over functions of many variables, part of which describe a sub-system and the rest account for its interaction with a random environment. The number of states of the sub-system with a given value of the objective function obeys a stretched exponential distribution, with exponent γ, and the interaction part is drawn at random from the same distribution, independently for each configuration of the whole system. Systems with γ = 1 then correspond to OLM, and we find that they sit at the boundary between two regions with markedly different properties. For all γ > 0 the system exhibits a freezing phase transition. The transition is discontinuous for γ < 1 and it is continuous for γ > 1. The region γ > 1 corresponds to learnable energy landscapes and the behaviour of the sub-system becomes predictable as the size of the environment exceeds a critical threshold. For γ < 1, instead, the energy landscape is unlearnable and the behaviour of the system becomes more and more unpredictable as the size of the environment increases. Sub-systems with γ = 1 (OLM) feature a behaviour which is independent of the relative size of the environment. This is consistent with the expectation that efficient representations should be largely independent of the level of detail of the description of the environment. arXiv:1904.09144v2 [physics.data-an]