On Data-Processing and Majorization Inequalities for f-Divergences with Applications

Abstract

This paper is focused on derivations of data-processing and majorization inequalities for f -divergences, and their applications in information theory and statistics. For the accessibility of the material, the main results are first introduced without proofs, followed by exemplifications of the theorems with further related analytical results, interpretations, and information-theoretic applications. One application refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds on the list decoding error probability are reproduced in a unified way, and new bounds are obtained and exemplified numerically. Another application is related to a study of the quality of approximating a probability mass function, induced by the leaves of a Tunstall tree, by an equiprobable distribution. The compression rates of finite-length Tunstall codes are further analyzed for asserting their closeness to the Shannon entropy of a memoryless and stationary discrete source. Almost all the analysis is relegated to the appendices, which form the major part of this manuscript.

Difference equations related to majorization theorems via Montgomery identity and Green’s functions with application to the Shannon entropy

“…The results we obtain in this paper are closely related to the contents given in [1][2][3][4][5]. Moreover, some related results with the present topic can also be found in [10,11,27,41,42].…”

“…If we reverse the sign of inequalities in (41) and (43), then inequalities (42) and (44) are also reversed.…”

“…Theorem 10 Let all the assumptions of Theorem 6 be true. If f is n-convex, where n is even, then inequalities (42) and (44) hold.…”

On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid

“…Since the vector-skew Jensen divergence is an f-divergence, we easily obtain Fano and Pinsker inequalities following [ 32 ], or reverse Pinsker inequalities following [ 33 , 34 ] (i.e., upper bounds for the vector-skew Jensen divergences using the total variation metric distance), data processing inequalities using [ 35 ], etc.…”

Generalizations of Fano’s Inequality for Conditional Information Measures via Majorization Theory

“…Polyanskiy–Verdú [ 57 ] showed a lower bound on Sibson’s -mutual information by using the data processing lemma for the Rényi divergence. Recently, Sason [ 58 ] generalized Fano’s inequality with list decoding via the strong data processing lemma for the f -divergences.…”