Potential conditional mutual information: Estimators and properties

Rahimzamani, Arman; Kannan, Sreeram

doi:10.7287/peerj.preprints.3345

Cited by 3 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using a similar technique to estimate conditional mutual information, Frenzel and Pompe (FP) first, though several other papers as well [17], [18], [19], [20], [21] used…”

Section: Frenzel and Pompe Estimator Of Conditional Mutual Informationmentioning

confidence: 99%

Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors

Mesner,

Shalizi

2019

Preprint

View full text Add to dashboard Cite

Fields like public health, public policy, and social science often want to quantify the degree of dependence between variables whose relationships take on unknown functional forms. Typically, in fact, researchers in these fields are attempting to evaluate causal theories, and so want to quantify dependence after conditioning on other variables that might explain, mediate or confound causal relations. One reason conditional mutual information is not more widely used for these tasks is the lack of estimators which can handle combinations of continuous and discrete random variables, common in applications. This paper develops a new method for estimating mutual and conditional mutual information for data samples containing a mix of discrete and continuous variables. We prove that this estimator is consistent and show, via simulation, that it is more accurate than similar estimators.

show abstract

“…Using a similar technique to estimate conditional mutual information, Frenzel and Pompe (FP) first, though several other papers as well [17], [18], [19], [20], [21] used…”

Section: Frenzel and Pompe Estimator Of Conditional Mutual Informationmentioning

confidence: 99%

Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors

Mesner,

Shalizi

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…2 simultaneously. First, CMI can measure the nonlinear dependency (Rahimzamani and Kannan, 2017) between two random variables, making it a competent statistic to interpret nonlinear CCA variants. Second, the conditional independent constraint for CCA-based models is automatically satisfied with the minimum CMI, since the optimal, I(X; Y |Z) = 0 (Cover and Thomas, 2012), is achieved with Eq.…”

Section: Minimum Cmi: a New Criterion For Ccamentioning

confidence: 99%

Probabilistic CCA with Implicit Distributions

Shi¹,

Pan²,

Xu³

et al. 2019

Preprint

View full text Add to dashboard Cite

Canonical Correlation Analysis (CCA) is a classic technique for multi-view data analysis. To overcome the deficiency of linear correlation in practical multi-view learning tasks, various CCA variants were proposed to capture nonlinear dependency. However, it is non-trivial to have an in-principle understanding of these variants due to their inherent restrictive assumption on the data and latent code distributions. Although some works have studied probabilistic interpretation for CCA, these models still require the explicit form of the distributions to achieve a tractable solution for the inference. In this work, we study probabilistic interpretation for CCA based on implicit distributions. We present Conditional Mutual Information (CMI) as a new criterion for CCA to consider both linear and nonlinear dependency for arbitrarily distributed data. To eliminate direct estimation for CMI, in which explicit form of the distributions is still required, we derive an objective which can provide an estimation for CMI with efficient inference methods. To facilitate Bayesian inference of multi-view analysis, we propose Adversarial CCA (ACCA), which achieves consistent encoding for multiview data with the consistent constraint imposed on the marginalization of the implicit posteriors. Such a model would achieve superiority in the alignment of the multi-view data with implicit distributions. It is interesting to note that most of the existing CCA variants can be connected with our proposed CCA model by assigning specific form for the posterior and likelihood distributions. Extensive experiments on nonlinear † indicates the corresponding author.

show abstract

“…4. Directed information measures the amount of information between two random processes [14,15] and is shown as the correct metric in identifying time-series graphical models [16][17][18][19][20][21].…”

Section: Conditional Mutual Information Measures the Amount Of Inform...mentioning

confidence: 99%

Estimators for Multivariate Information Measures in General Probability Spaces

Rahimzamani,

Asnani,

Viswanath

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Information theoretic quantities play an important role in various settings in machine learning, including causality testing, structure inference in graphical models, time-series problems, feature selection as well as in providing privacy guarantees. A key quantity of interest is the mutual information and generalizations thereof, including conditional mutual information, multivariate mutual information, total correlation and directed information. While the aforementioned information quantities are well defined in arbitrary probability spaces, existing estimators add or subtract entropies (we term them ΣH methods). These methods work only in purely discrete space or purely continuous case since entropy (or differential entropy) is well defined only in that regime.In this paper, we define a general graph divergence measure (GDM),as a measure of incompatibility between the observed distribution and a given graphical model structure. This generalizes the aforementioned information measures and we construct a novel estimator via a coupling trick that directly estimates these multivariate information measures using the Radon-Nikodym derivative. These estimators are proven to be consistent in a general setting which includes several cases where the existing estimators fail, thus providing the only known estimators for the following settings: (1) the data has some discrete and some continuous valued components (2) some (or all) of the components themselves are discrete-continuous mixtures (3) the data is real-valued but does not have a joint density on the entire space, rather is supported on a low-dimensional manifold. We show that our proposed estimators significantly outperform known estimators on synthetic and real datasets.

show abstract

Potential conditional mutual information: Estimators and properties

Cited by 3 publications

References 13 publications

Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors

Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors

Probabilistic CCA with Implicit Distributions

Estimators for Multivariate Information Measures in General Probability Spaces

Contact Info

Product

Resources

About