Bayesian Error-Based Sequences of Statistical Information Bounds

Prasad, Sudhakar

doi:10.1109/tit.2015.2457913

Cited by 8 publications

(10 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the special case of finite alphabets, this result was obtained in [25], [48] and [76]. In the finite alphabet setting, Prasad [58,Section 5] recently refined the bound in (181) by lower bounding H(X|Y ) subject to the knowledge of the first two largest posterior probabilities rather than only the largest one; following the same approach, [58, Section 6] gives a refinement of Fano's inequality. Example 1 (cont.)…”

Section: (180)mentioning

confidence: 99%

Arimoto–Rényi Conditional Entropy and Bayesian $M$ -Ary Hypothesis Testing

Sason

Verdú

2018

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

This paper gives upper and lower bounds on the minimum error probability of Bayesian Mary hypothesis testing in terms of the Arimoto-Rényi conditional entropy of an arbitrary order α. The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy (α = 1) is demonstrated. In particular, in the case where M is finite, we show how to generalize Fano's inequality under both the conventional and list-decision settings. As a counterpart to the generalized Fano's inequality, allowing M to be infinite, a lower bound on the Arimoto-Rényi conditional entropy is derived as a function of the minimum error probability. Explicit upper and lower bounds on the minimum error probability are obtained as a function of the Arimoto-Rényi conditional entropy for both positive and negative α. Furthermore, we give upper bounds on the minimum error probability as functions of the Rényi divergence. In the setup of discrete memoryless channels, we analyze the exponentially vanishing decay of the Arimoto-Rényi conditional entropy of the transmitted codeword given the channel output when averaged over a random-coding ensemble. Index TermsArimoto-Rényi conditional entropy, Bayesian minimum probability of error, Chernoff information, Fano's inequality, list decoding, M -ary hypothesis testing, random coding, Rényi entropy, Rényi divergence. I. Sason is with the Andrew and Erna Viterbi

show abstract

Section: (180)mentioning

confidence: 99%

Arimoto–Rényi Conditional Entropy and Bayesian $M$ -Ary Hypothesis Testing

Sason

Verdú

2018

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…However, when more constraints on the the joint distribution of X and Y are given, tighter arXiv:1812.01475v2 [cs.IT] 11 Jan 2019 bounds can be obtained. Prasad [8] introduced two series of lower bounds on H(X|Y ) based on partial knowledge of the posterior distribution p(x|y). The first is in terms of the k largest posterior probabilities p(x|y) for each y, that we could label p 1 (y), p 2 (y), .…”

Section: A Equivocation Mutual Information and The Minimal Probabilmentioning

confidence: 99%

A Tight Upper Bound on Mutual Information

Hledík

Sokolowski

Tkačik

2019

2019 IEEE Information Theory Workshop (ITW)

View full text Add to dashboard Cite

We derive a tight lower bound on equivocation (conditional entropy), or equivalently a tight upper bound on mutual information between a signal variable and channel outputs. The bound is in terms of the joint distribution of the signals and maximum a posteriori decodes (most probable signals given channel output). As part of our derivation, we describe the key properties of the distribution of signals, channel outputs and decodes, that minimizes equivocation and maximizes mutual information. This work addresses a problem in data analysis, where mutual information between signals and decodes is sometimes used to lower bound the mutual information between signals and channel outputs. Our result provides a corresponding upper bound.

show abstract

“…Using this, Equation 34 can be reexpressed as For discrete parameters, which are not differentiable and therefore do not have a defined Fisher information matrix, Fano's inequality provides a similar lower bound relating Lindley information and estimation error (e.g., Cover & Thomas, 1991;Prasad, 2015):…”

Section: The Relation In Equation 26mentioning

confidence: 99%

“…Interestingly, tight upper bounds to the Lindley information are also available for discrete variables (that is, reversing the direction of the inequality in Equation 37; Feder & Merhav, 1994;Prasad, 2015). This suggests that an even more precise global bounds to the reliability might be obtained, by applying upper as well as lower bounds.…”

Section: Global Minimax Bounds On Reliability and Measurement Potentialmentioning

confidence: 99%

Reliability as Lindley Information

Ke¹

2017

Preprint

View full text Add to dashboard Cite

This paper introduces a generalized definition of reliability based on Lindley information, which is the mutual information between an observed measure and latent attribute. This definition reduces to the traditional definition of reliability in the case of normal variables, but can be applied to any joint distribution of observed and latent variables. Importantly, unlike traditional definitions of reliability, this formulation of reliability applies to vector-or matrix-valued estimates and summaries of responses, and therefore generalizes reliability to sets of scores and estimates in addition to individual scores and estimates. This formulation also leads to new bounds for reliability, as well as newly reported relationships between reliability and the traditional Fisher information function familiar in item response theory literature. The generalized reliability can be estimated using formulae, or methods used in Bayesian inference such as Markov Chain Monte Carlo (MCMC) depending on the case. Examples based on well-studied datasets are provided, as well as an application to randomly-varying intraindividual covariance structures.Running head: GENERALIZED RELIABILITY 3 The aim of this paper is to introduce a generalized definition of reliability given aswhere ι is the Lindley information (Lindley, 1956), which in this context is the mutual information between an observed variable X and a latent attribute θ:Either X or θ could be multidimensional.Lindley information can be interpreted various ways depending on inferential paradigm and perspective: as an index of the expected change in probability of X and θ under knowledge of their joint distribution, relative to knowledge only of their independent marginal distributions; as the expected change in probability of θ given the data, relative to the prior distribution of θ; as the expected per-observation log-likelihood ratio comparing a model of measure informativeness to a model of measure uninformativeness (Kinney & Atwal, 2014); or as the difference in codelength required to describe the data and θ under a model of informativeness compared to a model of uninformativeness.More generally, Lindley information can be interpreted as the divergence of the joint observed-latent distribution p(X, θ) from the independence distribution p(X)p(θ). As such, it is a special case of f-divergence (Amari, 2009;Gilardoni, 2010;Liese & Vajda, 2006;Sason & Verdu, 2016), which can be defined aswhere f(u) is a convex function f:(0, ∞) → , and p and q are probability densities absolutely continuous with ℝ regard to a reference measure ν. Lindley information is obtained by setting f(u) = u ln u, with p = p(X, θ) and q = p(X)p(θ), and is equivalent to the relative entropy or Kullback-Liebler divergence between p(X, θ) and p(X)p(θ).Like traditional reliability, the generalized definition of reliability implied by Equations 1 and 2 produces a statistic that generally ranges between 0 and 1, and reflects the measurement precision of an indicator, such that values closer to 1 indic...

show abstract

Bayesian Error-Based Sequences of Statistical Information Bounds

Cited by 8 publications

References 13 publications

Arimoto–Rényi Conditional Entropy and Bayesian $M$ -Ary Hypothesis Testing

Arimoto–Rényi Conditional Entropy and Bayesian $M$ -Ary Hypothesis Testing

A Tight Upper Bound on Mutual Information

Reliability as Lindley Information

Contact Info

Product

Resources

About