On f-Divergences: Integral Representations, Local Behavior, and Inequalities

Self Cite

This paper is focused on derivations of data-processing and majorization inequalities for f -divergences, and their applications in information theory and statistics. For the accessibility of the material, the main results are first introduced without proofs, followed by exemplifications of the theorems with further related analytical results, interpretations, and information-theoretic applications. One application refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds on the list decoding error probability are reproduced in a unified way, and new bounds are obtained and exemplified numerically. Another application is related to a study of the quality of approximating a probability mass function, induced by the leaves of a Tunstall tree, by an equiprobable distribution. The compression rates of finite-length Tunstall codes are further analyzed for asserting their closeness to the Shannon entropy of a memoryless and stationary discrete source. Almost all the analysis is relegated to the appendices, which form the major part of this manuscript.

Section: Illustration Of Theorem 7 and Further Resultsmentioning

confidence: 99%

“…Letting α * := max{α 1 , α 2 } gives the result in (66) for all α > α * . Item f) of Theorem 5 is a direct consequence of [62,Lemma 4], which relies on [50, Theorem 3]. Let g(t) := (t − 1) 2 for t ≥ 0 (hence, D g (· ·) is the χ 2 divergence).…”

Section: Appendix D Proof Of Theoremmentioning

confidence: 99%

On Data-Processing and Majorization Inequalities for f-Divergences with Applications

Sason

2019

Self Cite

“…Let P and Q be distributions defined on a common probability space that have densities p and q with respect to a dominating measure

. The relative entropy (or Kullback–Leibler divergence) is defined according to

and the chi-squared divergence is defined as

Both of these divergences can be seen as special cases of the general class of f -divergence measures and there exists a rich literature on comparisons between different divergences [ 8 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ]. The chi-squared divergence can also be viewed as the squared

distance between

and

.…”

Section: Bounds On Mutual Informationmentioning

confidence: 99%

A Two-Moment Inequality with Applications to Rényi Entropy and Mutual Information

Reeves

2020

This paper explores some applications of a two-moment inequality for the integral of the rth power of a function, where 0<r<1. The first contribution is an upper bound on the Rényi entropy of a random vector in terms of the two different moments. When one of the moments is the zeroth moment, these bounds recover previous results based on maximum entropy distributions under a single moment constraint. More generally, evaluation of the bound with two carefully chosen nonzero moments can lead to significant improvements with a modest increase in complexity. The second contribution is a method for upper bounding mutual information in terms of certain integrals with respect to the variance of the conditional density. The bounds have a number of useful properties arising from the connection with variance decompositions.

“…The α-Jensen–Shannon divergences are Csiszár f -divergences [ 22 , 23 , 24 ]. An f -divergence is defined for a convex function f , strictly convex at 1, and satisfies

as:

…”

Section: Introductionmentioning

confidence: 99%

On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid

Nielsen

2020

The Jensen-Shannon divergence is a renown bounded symmetrization of the Kullback-Leibler divergence which does not require probability densities to have matching supports. In this paper, we introduce a vector-skew generalization of the scalar α-Jensen-Bregman divergences and derive thereof the vector-skew α-Jensen-Shannon divergences. We study the properties of these novel divergences and show how to build parametric families of symmetric Jensen-Shannon-type divergences. Finally, we report an iterative algorithm to numerically compute the Jensen-Shannon-type centroids for a set of probability densities belonging to a mixture family: This includes the case of the Jensen-Shannon centroid of a set of categorical distributions or normalized histograms.