The Poincaré-Shannon Machine: Statistical Physics and Machine Learning Aspects of Information Cohomology

Abstract

This paper presents the computational methods of information cohomology applied to genetic expression in [125,16,17] and in the companion paper [16] and proposes its interpretations in terms of statistical physics and machine learning. In order to further underline the Hochschild cohomological nature af information functions and chain rules, following [13,14,133], the computation of the cohomology in low degrees is detailed to show more directly that the k multivariate mutual-informations (I k ) are k-coboundaries. The k-cocycles condition corresponds to I k = 0, which generalize statistical independence to arbitrary dimension k [16]. Hence the cohomology can be interpreted as quantifying the statistical dependences and the obstruction to factorization. The topological approach allows to investigate Shannon's information in the multivariate case without the assumptions of independent identically distributed variables and of statistical interactions without mean field approximations. We develop the computationally tractable subcase of simplicial information cohomology represented by entropy H k and information I k landscapes and their respective paths. The I1 component defines a self-internal energy functional U k , and (−1) k I k,k≥2 components define the contribution to a free energy functional G k (the total correlation) of the k-body interactions. The set of information paths in simplicial structures is in bijection with the symmetric group and random processes, provides a trivial topological expression of the 2nd law of thermodynamic. The local minima of free-energy, related to conditional information negativity, and conditional independence, characterize a minimum free energy complex. This complex formalizes the minimum free-energy principle in topology, provides a definition of a complex system, and characterizes a multiplicity of local minima that quantifies the diversity observed in biology. I give an interpretation of this complex in terms of frustration in glass and of Van Der Walls k-body interactions for data points.

Elements of qualitative cognition: An information topology perspective

“…The main theorems, definitions and data analysis [147,134,133] establish the following results, here included with comments about their relevance regarding consciousness and neural processing theories.…”

“…It appears by direct computation in this cohomology that mutual informations with an odd number of variables are minus the coboundary of even degrees ∂ 2k F = −I 2k+1 . Obtaining even mutual informations is achieved by introducing a second coboundary with either trivial or symmetric action [132,89,133], giving the even mutual-informations as minus the odd symmetric coboundaries ∂ 2k−1 * F = −I 2k . The independence of two variables (I 2 = 0) is then directly generalized to k-variables and gives the cocycles I k = 0 [134].…”

“…The third point of interest is that cohomology is the science of the forms (patterns) of spaces. Information topology hence provides a preliminary theory of the shapes of probabilistic structures on which it is possible to develop methods of pattern recognition-characterization for machine learning and the quantification of epigenetic landscapes for biological adaptive dynamics, following Waddington and Thom [137,138,133]. Concerning classification-recognition aspects, information structures can be considered as universal: as a partition is equivalent to an equivalence class all possible classification are represented in an information structure.…”

“…The random variables formalize neurons that are intrinsically probabilistic and possibly multivalued, generalizing binary and deterministic neurons such as McCulloch and Pitts' formal neurons. Information topology allows us to quantify the structure of statistical interactions within a set of empirical data and to express these interactions in terms of statistical physics, machine learning and epigenetic dynamics [133]. The combinatorics of general variable complexes being governed by Bell's numbers, their effective computation on classical computers is illusory.…”

The Design of Global Correlation Quantifiers and Continuous Notions of Statistical Sufficiency

“…The approach by Baez et al shows that a particular class of functors from the category FinStat, which is a finite set equipped with a probability distribution, are scalar multiples of the entropy [15]. The papers by Baudot et al [16][17][18] also take a category theoretical approach however their results are more focused on the topological properties of information theoretic quantities. Both Baez et al and Baudot et al discuss various information theoretic measures such as the relative entropy, mutual information, total correlation, and others.…”