Local Intrinsic Dimensionality, Entropy and Statistical Divergences

Abstract

Properties of data distributions can be assessed at both global and local scales. At a highly localized scale, a fundamental measure is the local intrinsic dimensionality (LID), which assesses growth rates of the cumulative distribution function within a restricted neighborhood and characterizes properties of the geometry of a local neighborhood. In this paper, we explore the connection of LID to other well known measures for complexity assessment and comparison, namely, entropy and statistical distances or divergences. In an asymptotic context, we develop analytical new expressions for these quantities in terms of LID. This reveals the fundamental nature of LID as a building block for characterizing and comparing data distributions, opening the door to new methods for distributional analysis at a local scale.

Correlations of Cross-Entropy Loss in Machine Learning

“…We rely heavily on work by Harremoës [ 5 ] and Topsøe [ 4 ], who show strong bounds among these functions; we extend that work here to show how these bounds give extremely strong correlations among high-dimensional embeddings. Other more recent related work includes [ 6 ], which shows a very strong convergence of the measured distribution of values as the locality over which the distances are measured tends towards infinitesimal.…”