Squared-Loss Mutual Information via High-Dimension Coherence Matrix Estimation

Abstract

Squared-loss mutual information (SMI) is a surrogate of Shannon mutual information that is more advantageous for estimation. On the other hand, the coherence matrix of a pair of random vectors, a power-normalized version of the sample cross-covariance matrix, is a well-known second-order statistic found in the core of fundamental signal processing problems, such as canonical correlation analysis (CCA). This paper shows that SMI can be estimated from a pair of independent and identically distributed (i.i.d.) samples as a squared Frobenius norm of a coherence matrix estimated after mapping the data onto some fixed feature space. Moreover, low computation complexity is achieved through the fast Fourier transform (FFT) by exploiting the Toeplitz structure of the involved autocorrelation matrices in that space. The performance of the method is analyzed via computer simulations using Gaussian mixture models.

Regularized Estimation of Information via High Dimensional Canonical Correlation Analysis

“…5) The proposal of a reduced complexity approximate estimator resorting to the asymptotic behavior of Toeplitz matrices. Concerning the related works, the presented approach is an extension of the main ideas shortly provided by the authors in [7]. Although the interest on surrogates of entropy, KL divergence and mutual information, such as Rényi entropy, Rényi divergence, f -divergence and chi-squared (χ 2 ) divergence has a long and rich history (see [3] and references therein), its use for data analytics has been particularly focused in [4].…”

“…KL divergence is non-negative, and it is zero if and only if p X (x) = q X (x). A natural surrogate of KL divergence can be obtained by applying the Jensen inequality to (7). In particular, as ln(.)…”

“…Using the Taylor expansion of ln((1 + α) −1 ) up to the second order, i.e. −α + α 2 /2 + O(α 3 ), we can write the KL divergence in (7) as…”

“…Given the physical sense of the proposed regularization, we propose (see [7]) a uniform symmetric and finite sampling of ω 1 and ω 2 to define mappings φ X (.) : R → C N and φ Y (.)…”

Estimation of Information in Parallel Gaussian Channels via Model Order Selection

“…This line of research finds numerous applications in data science and machine learning. Recently, the problem of estimating information has been linked in [2] with the aforementioned problem of coherence estimation by mapping the bivariate data onto a high-dimensional feature space based on the empirical characteristic function. In particular, the Frobenius norm of the coherence matrix computed after this high-dimensional mapping converges with M to the so-called squared-loss mutual information [14].…”