A Quick and Easy Way to Estimate Entropy and Mutual Information for Neuroscience

Abstract

Calculations of entropy of a signal or mutual information between two variables are valuable analytical tools in the field of neuroscience. They can be applied to all types of data, capture non-linear interactions and are model independent. Yet the limited size and number of recordings one can collect in a series of experiments makes their calculation highly prone to sampling bias. Mathematical methods to overcome this so-called “sampling disaster” exist, but require significant expertise, great time and computational costs. As such, there is a need for a simple, unbiased and computationally efficient tool for estimating the level of entropy and mutual information. In this article, we propose that application of entropy-encoding compression algorithms widely used in text and image compression fulfill these requirements. By simply saving the signal in PNG picture format and measuring the size of the file on the hard drive, we can estimate entropy changes through different conditions. Furthermore, with some simple modifications of the PNG file, we can also estimate the evolution of mutual information between a stimulus and the observed responses through different conditions. We first demonstrate the applicability of this method using white-noise-like signals. Then, while this method can be used in all kind of experimental conditions, we provide examples of its application in patch-clamp recordings, detection of place cells and histological data. Although this method does not give an absolute value of entropy or mutual information, it is mathematically correct, and its simplicity and broad use make it a powerful tool for their estimation through experiments.

Tidal frequencies and quasiperiodic subsurface water level variations dominate redox dynamics in a salt marsh system

“…Further, we normalised I(X;Y) to scale the results between 0 (no mutual information) and 1 (perfect correlation). The normalisation uses the entropy H(X) of each individual signal, and can be calculated as (Kvålseth, 2017; Zbili & Rama, 2021): And H(X) of the discrete random variable X is calculated from its probability ( P[x] ) and surprise ( logP[x] ) as: …”

“…Further, we normalised I(X;Y) to scale the results between 0 (no mutual information) and 1 (perfect correlation). The normalisation uses the entropy H(X) of each individual signal, and can be calculated as (Kvålseth, 2017;Zbili & Rama, 2021):…”

Cognitive engagement on social media: A study of the effects of visual cueing in educational videos

“…A video's entropy value was first calculated based on the sampled images from the video. As entropy values of images stay quite consistent across the video (Zbili & Rama, 2021), we used six equally spaced images to approximate a video's entropy. A total of 1,176 images were extracted from the 196 videos.…”

Entropy, Economics, and Criticality

“…The first point is simply one that we may need to accept, even as computers become faster, it will likely remain the case that, for example, the Pearson correlation will be faster to compute than the corresponding MI. However, the computations will get faster in absolute terms as computers get faster, and other fields with large datasets, such as neuroscience, have seen the benefits of these new methods [ 43 , 44 ] with efficiency gains being made as well [ 45 , 46 ]. On the other hand software packages are becoming more readily available, and economics can benefit from the software advances that have been made in other fields.…”