Mutual Information, Fisher Information, and Population Coding

Abstract

In the context of parameter estimation and model selection, it is only quite recently that a direct link between the Fisher information and information-theoretic quantities has been exhibited. We give an interpretation of this link within the standard framework of information theory. We show that in the context of population coding, the mutual information between the activity of a large array of neurons and a stimulus to which the neurons are tuned is naturally related to the Fisher information. In the light of this result, we consider the optimization of the tuning curves parameters in the case of neurons responding to a stimulus represented by an angular variable.

Retarded Learning: Rigorous Results from Statistical Mechanics

“…The asymptotics can be found analytically by expanding both bounds (7) and (9) for q → 1. For a smooth potential V both bounds give asymptotically the same logarithmic growth R Bayes m /N ≃ 1/2 ln α which can also be obtained by well known asymptotic expansions involving the Fisher information matrix [22,23,11]. On the other hand, our bounds can also be used when these standard asymptotic expansions do not apply, e.g.…”

“…We combine information theoretic bounds for the performance of statistical estimators (see e.g. [10,11,12]) with an elementary variational principle of statistical physics [13]. This will allow us to compute rigorous upper and lower bounds for the critical number of examples at which a transition occurs.…”

Information capacity in the weak-signal approximation

“…The already mentioned low-noise approximation to MI is constructed by employing the Cramer-Rao bound [12,[21][22][23]]. Although we demonstrate that the high-noise approximation also involves FI, we never employ the Cramer-Rao bound and the appearance of FI is due to certain asymptotic properties of the KL distance [28].…”

Optimal computation with attractor networks

“…(26), (2) compute the derivative of the log likelihood, Eq. (28), and (3) show that the maximum likelihood estimator is unbiased and efficient (i.e., it reaches the Cramer-Rao bound).…”

“…We use the log of the covariance matrix to asses the quality of the estimator because it determines, to a large extent, the mutual information between the noisy neuronal responses, a, and the stimulus, s: the smaller the determinant of the covariance matrix, the larger the mutual information [3]. (Strictly speaking, this result applies only when the neurons are uncorrelated.…”