Information-theoretic asymptotics of Bayes methods

Clarke, Bertrand; Barron, Andrew R.

doi:10.1109/18.54897

Cited by 379 publications

(374 citation statements)

References 40 publications

Supporting

Mentioning

357

Contrasting

Order By: Relevance

“…The already mentioned low-noise approximation to MI is constructed by employing the Cramer-Rao bound [12,[21][22][23]]. Although we demonstrate that the high-noise approximation also involves FI, we never employ the Cramer-Rao bound and the appearance of FI is due to certain asymptotic properties of the KL distance [28].…”

Section: Measures Of Informationmentioning

confidence: 99%

Information capacity in the weak-signal approximation

Kostal

2010

Phys. Rev. E

View full text Add to dashboard Cite

We derive an approximate expression for mutual information in a broad class of discrete-time stationary channels with continuous input, under the constraint of vanishing input amplitude or power. The approximation describes the input by its covariance matrix, while the channel properties are described by the Fisher information matrix. This separation of input and channel properties allows us to analyze the optimality conditions in a convenient way. We show that input correlations in memoryless channels do not affect channel capacity since their effect decreases fast with vanishing input amplitude or power. On the other hand, for channels with memory, properly matching the input covariances to the dependence structure of the noise may lead to almost noiseless information transfer, even for intermediate values of the noise correlations. Since many model systems described in mathematical neuroscience and biophysics operate in the high noise regime and weak-signal conditions, we believe, that the described results are of potential interest also to researchers in these areas.

show abstract

Section: Measures Of Informationmentioning

confidence: 99%

Information capacity in the weak-signal approximation

Kostal

2010

Phys. Rev. E

View full text Add to dashboard Cite

show abstract

“…The asymptotics can be found analytically by expanding both bounds (7) and (9) for q → 1. For a smooth potential V both bounds give asymptotically the same logarithmic growth R Bayes m /N ≃ 1/2 ln α which can also be obtained by well known asymptotic expansions involving the Fisher information matrix [22,23,11]. On the other hand, our bounds can also be used when these standard asymptotic expansions do not apply, e.g.…”

mentioning

confidence: 68%

Retarded Learning: Rigorous Results from Statistical Mechanics

Herschkowitz¹,

Opper

2001

Phys. Rev. Lett.

View full text Add to dashboard Cite

We study learning of probability distributions characterized by an unknown symmetry direction. Based on an entropic performance measure and the variational method of statistical mechanics we develop exact upper and lower bounds on the scaled critical number of examples below which learning of the direction is impossible. The asymptotic tightness of the bounds suggests an asymptotically optimal method for learning nonsmooth distributions.

show abstract

“…The maximum is always attained in (10) since for each at most a finite number of elements fulfill . Observe immediately the correspondence in terms of description lengths rather than probabilities Then the MDL principle is obvious: minimizes the joint description length of the model plus the data given the model 1 (see (8) and (9)). As explained before, we stick to the product notation.…”

Section: MDL Estimator and Predictionsmentioning

confidence: 99%

“…The converse holds as well. We introduce the abbreviation (8) for a semimeasure and and for the Bayes mixture . It is common to ignore the somewhat irrelevant restriction that code lengths must be an integer.…”

Section: Prerequisites and Notationmentioning

confidence: 99%

Asymptotics of Discrete MDL for Online Prediction

Poland

Hutter

2005

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Abstract-Minimum description length (MDL) is an important principle for induction and prediction, with strong relations to optimal Bayesian learning. This paper deals with learning processes which are independent and identically distributed (i.i.d.) by means of two-part MDL, where the underlying model class is countable. We consider the online learning framework, i.e., observations come in one by one, and the predictor is allowed to update its state of mind after each time step. We identify two ways of predicting by MDL for this setup, namely, a static and a dynamic one. (A third variant, hybrid MDL, will turn out inferior.) We will prove that under the only assumption that the data is generated by a distribution contained in the model class, the MDL predictions converge to the true values almost surely. This is accomplished by proving finite bounds on the quadratic, the Hellinger, and the Kullback-Leibler loss of the MDL learner, which are, however, exponentially worse than for Bayesian prediction. We demonstrate that these bounds are sharp, even for model classes containing only Bernoulli distributions. We show how these bounds imply regret bounds for arbitrary loss functions. Our results apply to a wide range of setups, namely, sequence prediction, pattern classification, regression, and universal induction in the sense of algorithmic information theory among others.

show abstract

Information-theoretic asymptotics of Bayes methods

Cited by 379 publications

References 40 publications

Information capacity in the weak-signal approximation

Information capacity in the weak-signal approximation

Retarded Learning: Rigorous Results from Statistical Mechanics

Asymptotics of Discrete MDL for Online Prediction

Contact Info

Product

Resources

About