Nonparametric estimation of Fisher information from real data

Abstract

The Fisher information matrix (FIM) is a widely used measure for applications including statistical inference, information geometry, experiment design, and the study of criticality in biological systems. The FIM is defined for a parametric family of probability distributions and its estimation from data follows one of two paths: either the distribution is assumed to be known and the parameters are estimated from the data or the parameters are known and the distribution is estimated from the data. We consider the latter case which is applicable, for example, to experiments where the parameters are controlled by the experimenter and a complicated relation exists between the input parameters and the resulting distribution of the data. Since we assume that the distribution is unknown, we use a nonparametric density estimation on the data and then compute the FIM directly from that estimate using a finite-difference approximation to estimate the derivatives in its definition. The accuracy of the estimate depends on both the method of nonparametric estimation and the difference Δθ between the densities used in the finite-difference formula. We develop an approach for choosing the optimal parameter difference Δθ based on large deviations theory and compare two nonparametric density estimation methods, the Gaussian kernel density estimator and a novel density estimation using field theory method. We also compare these two methods to a recently published approach that circumvents the need for density estimation by estimating a nonparametric f divergence and using it to approximate the FIM. We use the Fisher information of the normal distribution to validate our method and as a more involved example we compute the temperature component of the FIM in the two-dimensional Ising model and show that it obeys the expected relation to the heat capacity and therefore peaks at the phase transition at the correct critical temperature.

Fisher transfer entropy: quantifying the gain in transient sensitivity

“…Fisher information was analytically shown to diverge at the critical point approaching from one side (cf. empirical results of another study confirming this derivation [44]), and staying zero on the other side. We followed with a detailed analysis of FTE and demonstrated its divergence at the critical point, approaching from the same side where Fisher information itself is actually zero.…”

Information geometric analysis of phase transitions in complex patterns: the case of the Gray-Scott reaction–diffusion model

“…We used the Python package SimpleCV for the binarization and blob detection of the images and scipy.stats.gaussian_kde for the computation of the PDF from the blob sizes. The computation of the Fisher information from the PDF followed the description in [49] and the code for this computation is available online at [55].…”

“…Following [30] we set the expression under the integration to zero whenever any of the densities was zero for a given x. An extended discussion about the use of non-parametric density estimation (such as Kernel Density Estimations) and using finite dierence schemes in the computation of the Fisher information is given in [49].…”

Training Echo State Networks with Regularization Through Dimensionality Reduction

“…We believe that this work could be useful not only to enhance the prediction capabilities of an ESN, but also provide a new instrument for analysis of dynamical systems. As a follow-up of a recent work focused on identifying the edge of criticality of an ESN by evaluating the Fisher information on the state matrix [38], we plan to study the criticality using more reliable Fisher Information Matrix estimators, which are capable of working only on space with few dimensions (e.g., [25]). We also plan on investigating other dimensionality reduction methods, manifold learning and semi-supervised learning approaches to shrink and regularize the output of the network recurrent layer [4,5].…”