A Tight Upper Bound on Mutual Information

Abstract

We derive a tight lower bound on equivocation (conditional entropy), or equivalently a tight upper bound on mutual information between a signal variable and channel outputs. The bound is in terms of the joint distribution of the signals and maximum a posteriori decodes (most probable signals given channel output). As part of our derivation, we describe the key properties of the distribution of signals, channel outputs and decodes, that minimizes equivocation and maximizes mutual information. This work addresses a problem in data analysis, where mutual information between signals and decodes is sometimes used to lower bound the mutual information between signals and channel outputs. Our result provides a corresponding upper bound.

Estimating information in time-varying signals

“…For discrete inputs, classic work in information theory proved a number of upper bounds on this gap when the channel is known [46], with the Feder-Merhav bound perhaps being the most well known [34]; Feder-Merhav provides an upper bound on the channel capacity given the overall probability of error in MAP decoding. In a separate work [35], we computed a new upper bound on information I UB (U ; X) that is consistent with not just the overall probability of error as in Feder-Merhav bound, but with the full confusion matrix obtained from optimal MAP decoding, and showed that the new bound is tight.…”

“…This decoding gives the lowest average probability of error and the corresponding information lower bound can be used as a benchmark for information estimates derived from other model-free decoding approaches (that have at least the error probability of the MAP decoder); in Section 2.6 we compare Support Vector Machine (SVM), Gaussian Decoding (GD) and Neural Network (NN) decoding approaches. Upper bounds like the Feder-Merhav bound [34] and our improvement on it [35] complete the picture by estimating the gap between optimal decoding-derived and exact information values (Section 2.5).…”

“…Here, parameters ω consist of conditional means and (possibly regularized) covariance matrices of the Gaussian distributions that need to be estimated from data, following the test/train procedure analogous to SVM decoding. This method can be used with different parametric multivariate probability density functions replacing the multivariate Gaussian in Eq (35), with choices that approximate the statistics of the data (and thus the CME-derived distribution) better providing tighter lower bound estimates, I GD ( Û ; U ), of the exact information. By analogy with the exact MAP decoding using CME-derived response distribution, this method can also be understood as maximum a posteriori decoder but using approximate response distributions that need to be estimated from data.…”

Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation

“…where I min and I max are pre-defined constant values between zero and the maximum mutual information between x and y given any joint distribution p(x, y) ∈ P XY whose marginals satisfy x p(x, y) = p(y) and y p(x, y) = q(x). Hledík et al (2019) prove that the maximum mutual information max…”

Quantifying information of intracellular signaling: progress with machine learning

“…To overcome this issue, various machine-learning models, such as neural networks, can be used for classifiers to improve estimates [99]. In addition to the lower bound, an upper bound on the mutual information was derived [135].…”