The Statistical Minkowski Distances: Closed-Form Formula for Gaussian Mixture Models

Abstract

The traditional Minkowski distances are induced by the corresponding Minkowski norms in real-valued vector spaces. In this work, we propose novel statistical symmetric distances based on the Minkowski's inequality for probability densities belonging to Lebesgue spaces. These statistical Minkowski distances admit closed-form formula for Gaussian mixture models when parameterized by integer exponents. This result extends to arbitrary mixtures of exponential families with natural parameter spaces being cones: This includes the binomial, the multinomial, the zero-centered Laplacian, the Gaussian and the Wishart mixtures, among others. We also derive a Minkowski's diversity index of a normalized weighted set of probability distributions from Minkowski's inequality.Cauchy-Schwarz (CS) divergence [18,20], and a statistical distance based on discrete optimal transport [22,38].A distance D : X × X → R is a non-negative real-valued function D on the product space X × X such that D(p, q) = D((p, q)) = 0 iff. p = q. A distance D(p : q) between p and q may not be symmetric: This fact is emphasized by the ':' delimiter notation: D(p : q) = D(q : p). For example, the KL divergence is an oriented distance: KL(p : q) = KL(q : p). Two usual symmetrizations of the KL divergence are the Jeffreys' divergence and the Jensen-Shannon divergence [27]. Informally speaking, a divergence 2 is a smooth distance 3 that allows one to define an information-geometric structure [2]. In other words, a divergence is a smooth premetric distance [9].Recently, the Cauchy-Schwarz divergence [18] has been generalized to Hölder divergences [39]. These Cauchy and Hölder distances D(p : q) are said to be projective because D(λp : λ ′ q) = D(p : q) for any λ, λ ′ > 0. An important family of projective divergences for robust statistical inference are the γ-divergences [13,33]. Interestingly, those projective distances do not require to handle normalized probability densities but only need to consider positive densities instead (handy in applications). The Hölder projective divergences do not admit closed-form formula for GMMs, except for the very special case of the CS divergence. The underlying reason is that the conjugate exponents 1 α + 1 β = 1 of Hölder divergences would need to be both integers. This constraint yields α = β = 1, giving the special case of the CS divergence (the other integer exponent case is in the limit when α = 0 and β = ∞).

On a Variational Definition for the Jensen-Shannon Symmetrization of Distances Based on the Information Radius

“…Now, assume that is an arbitrary integer, and let us apply the binomial expansion for in the spirit of [ 46 , 47 ]: …”

Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities

“…Another approach when dealing with mixtures consists in designing new types of divergences which admit closed-form expressions. See [44,59,64] for some examples.…”

On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means

“…When considering exponential families, choose the weighted geometric mean for the abstract mean : , for . Indeed, it is well-known that the normalized weighted product of distributions belonging to the same exponential family also belongs to this exponential family [ 45 ]: where the normalization factor is for the skew Jensen divergence defined by: …”