Inequalities between entropy and index of coincidence derived from information diagrams

Abstract

To any discrete probability distribution P we can associate its entropy H(P ) = − p i ln p i and its index of coincidence IC(P ) = p 2 i . The main result of the paper is the determination of the precise range of the map P (IC(P ), H(P )). The range looks much like that of the map P (P max , H(P )) where P max is the maximal point probability, cf. research from 1965 (Kovalevskij [18]) to 1994 (Feder and Merhav [7]). The earlier results, which actually focus on the probability of error 1 − P max rather than P max , can be conceived as limiting cases of results obtained by methods here presented. Ranges of maps as those indicated are called Information Diagrams.The main result gives rise to precise lower as well as upper bounds for the entropy function. Some of these bounds are essential for the exact solution of certain problems of universal coding and prediction for Bernoulli sources. Other applications concern Shannon theory (relations betweeen various measures of divergence), statistical decision theory and rate distortion theory.Two methods are developed. One is topological, another involves convex analysis and is based on a "lemma of replacement" which is of independent interest in relation to problems of optimization of mixed type (concave/convex optimization).

Highly symmetric POVMs and their informational power

“…The problem of finding the minimum and maximum of the Shannon entropy under assumption that the index of coincidence is constant has been analyzed in [57] (some generalizations and related topics can be found also in [16,137]). By [57, Theorem 2.5.…”

Observable estimation of entanglement of formation and quantum discord for bipartite mixed quantum states

“…Actually, this problem is equivalent to seek the maximal and minimal H( µ) = − i µ i log 2 µ i for a given τ = 2(1 − i µ 2 i ), and the later one is a classical problem which has been solved in Refs. [48,49]. We use…”

Entropy, some new lower bounds

“…In Harremoës and Topsøe [1] the exact range of the map P (IC(P ), H(P )) with P varying over either M 1 + (n) or M 1 + (N) was determined. The ranges in question, termed IC/H-diagrams, were denoted ∆, respectively ∆ n :…”

“…Figure 1 illustrates the situation for the restricted diagrams ∆ n . The key result of [1] states that ∆ n is the bounded region determinated by a certain Jordan curve determined by n smooth arcs, the "upper arc" connecting Q 1 and Q n and then n − 1 "lower arcs" connecting Q n with Q n−1 , Q n−1 with Q n−2 etc. until Q 2 which is connected with Q 1 .…”

“…In [1] the main result was used to develop concrete upper bounds for the entropy function. Here our concern will be lower bounds.…”