The Differential Entropy of Mixtures: New Bounds and Applications

Strongly Convex Divergences

“…The following appears as Theorem III.1 in the preprint [14]. It states that skewing an fdivergence preserves its status as such.…”

“…A proof is given in the appendix for the convenience of the reader. Theorem 3.1 (Melbourne et al [14]). For t, s ∈ [0, 1] and an f -divergence, D f (•||•), in the sense that…”

“…It is based on a diffential relationship between the skew divergence [10] and the [8], see [13,16]. Lemma 3.5 (Theorem III.1 [14]). For P and Q probability measures, and t ∈ [0, 1] S 0,t (P ||Q) ≤ − log t|P − Q| T V .…”

“…Note that since ᾱi is the w average of the α j terms with α i removed, ᾱi ∈ [0, 1] and thus A ≤ 1. We will need the following Theorem from [14] for the upper bound. Theorem 3.3 ([14] Theorem 1.1).…”

Tight Lower Bound on Differential Entropy for Mixed Gaussian Distributions

“…In [19], the authors used the principle of entropy concavity deficit to introduce bounds on the differential entropy of a Gaussian mixture. This principle uses the difference between the differential entropy of the mixture and the weighted sum of differential entropies of its constituent components.…”