Calculation of Differential Entropy for a Mixed Gaussian Distribution

Michalowicz, Joseph V.; Nichols, Jonathan M.; Bucholtz, F.

doi:10.3390/entropy-e10030200

Cited by 57 publications

(52 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To complement our analysis above using L, we now look at how the (Gibbs) differential entropy S(t) = − dx p(x,t) ln p(x,t) (e.g., see [50], and using units where the Boltzmann constant K B = 1) changes during the phase transition. It is important to note that S differs from L in that it only depends on p at any instant in time, but not on the evolution that led to that PDF.…”

Section: Entropymentioning

confidence: 99%

Geometric structure and information change in phase transitions

Kim

Hollerbach

2017

Phys. Rev. E

View full text Add to dashboard Cite

We propose a toy model for a cyclic order-disorder transition and introduce a geometric methodology to understand stochastic processes involved in transitions. Specifically, our model consists of a pair of forward and backward processes (FPs and BPs) for the emergence and disappearance of a structure in a stochastic environment. We calculate time-dependent probability density functions (PDFs) and the information length L, which is the total number of different states that a system undergoes during the transition. Time-dependent PDFs during transient relaxation exhibit strikingly different behavior in FPs and BPs. In particular, FPs driven by instability undergo the broadening of the PDF with a large increase in fluctuations before the transition to the ordered state accompanied by narrowing the PDF width. During this stage, we identify an interesting geodesic solution accompanied by the self-regulation between the growth and nonlinear damping where the time scale τ of information change is constant in time, independent of the strength of the stochastic noise. In comparison, BPs are mainly driven by the macroscopic motion due to the movement of the PDF peak. The total information length L between initial and final states is much larger in BPs than in FPs, increasing linearly with the deviation γ of a control parameter from the critical state in BPs while increasing logarithmically with γ in FPs. L scales as | ln D| and D −1/2 in FPs and BPs, respectively, where D measures the strength of the stochastic forcing. These differing scalings with γ and D suggest a great utility of L in capturing different underlying processes, specifically, diffusion vs advection in phase transition by geometry. We discuss physical origins of these scalings and comment on implications of our results for bistable systems undergoing repeated order-disorder transitions (e.g., fitness).

show abstract

Section: Entropymentioning

confidence: 99%

Geometric structure and information change in phase transitions

Kim

Hollerbach

2017

Phys. Rev. E

View full text Add to dashboard Cite

show abstract

“…It thus generalizes the bounds reported in [7] to GMMs with arbitrary variances that are not necessarily equal.…”

Section: Without Loss Of Generality Consider Gmms In the Formmentioning

confidence: 75%

“…Alternatively, we can also use the base-2 logarithm (log 2 x = log x log 2 ) and get the entropy expressed in bit units. Although the KL divergence is available in closed-form for many distributions (in particular as equivalent Bregman divergences for exponential families [5], see Appendix C), it was proven that the Kullback-Leibler divergence between two (univariate) GMMs is not analytic [6] (see also the particular case of a GMM of two components with the same variance that was analyzed in [7]). See Appendix A for an analysis.…”

Section: Introductionmentioning

confidence: 99%

Guaranteed Bounds on Information-Theoretic Measures of Univariate Mixtures Using Piecewise Log-Sum-Exp Inequalities

Nielsen

Sun

2016

Entropy

View full text Add to dashboard Cite

Information-theoretic measures, such as the entropy, the cross-entropy and the Kullback-Leibler divergence between two mixture models, are core primitives in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte Carlo stochastic integration, approximated or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy, the Kullback-Leibler and the α-divergences of mixtures. We illustrate the versatile method by reporting our experiments for approximating the Kullback-Leibler and the α-divergences between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures and Gamma mixtures.

show abstract

“…In a variety of systems, adaptation takes place by inflating or deflating information, so that the "right" balance is achieved. Certainly, given that it is possible to derive upper and lower bounds for the differential entropy of a PDF (e.g., [20]), it should be also possible to define analytical bounds for the complexity measures for the given PDF. However, for practical purposes, complexity measures are constrained by the selected range of the PDF parameters.…”

Section: Discussionmentioning

confidence: 99%

Measuring the Complexity of Continuous Distributions

Santamaría-Bonfil

Fernández

Gershenson

2016

Entropy

View full text Add to dashboard Cite

Abstract:We extend previously proposed measures of complexity, emergence, and self-organization to continuous distributions using differential entropy. Given that the measures were based on Shannon's information, the novel continuous complexity measures describe how a system's predictability changes in terms of the probability distribution parameters. This allows us to calculate the complexity of phenomena for which distributions are known. We find that a broad range of common parameters found in Gaussian and scale-free distributions present high complexity values. We also explore the relationship between our measure of complexity and information adaptation.

show abstract

Calculation of Differential Entropy for a Mixed Gaussian Distribution

Abstract: In this work, an analytical expression is developed for the differential entropy of a mixed Gaussian distribution. One of the terms is given by a tabulated function of the ratio of the distribution parameters

Cited by 57 publications

References 11 publications

Geometric structure and information change in phase transitions

Geometric structure and information change in phase transitions

Guaranteed Bounds on Information-Theoretic Measures of Univariate Mixtures Using Piecewise Log-Sum-Exp Inequalities

Measuring the Complexity of Continuous Distributions

Contact Info

Product

Resources

About