On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means

Nielsen, Frank

doi:10.3390/e21050485

Cited by 149 publications

(113 citation statements)

References 56 publications

Supporting

Mentioning

113

Contrasting

Order By: Relevance

“…We prove that by taking the middle ground, one can meaningfully quantify the causal effect with mutual information and conditional mutual information in an unconfounded setting. To this end, we employ the weighted Jensen-Shannon divergence [ 76 , 77 ], which is sensitive to more than just the first moment of a distribution, as a measure of difference between interventional distributions. We then show that SCE and ACE (Definitions 4 and 5) are equivalent to conditional mutual information and mutual information, respectively, when the difference of means is replaced with the Jensen-Shannon divergence.…”

Section: Proposed Methods For Causal Effect Identificationmentioning

confidence: 99%

See 1 more Smart Citation

Information Theoretic Causal Effect Quantification

Wieczorek

Röth

2019

Entropy

View full text Add to dashboard Cite

Modelling causal relationships has become popular across various disciplines. Most common frameworks for causality are the Pearlian causal directed acyclic graphs (DAGs) and the Neyman-Rubin potential outcome framework. In this paper, we propose an information theoretic framework for causal effect quantification. To this end, we formulate a two step causal deduction procedure in the Pearl and Rubin frameworks and introduce its equivalent which uses information theoretic terms only. The first step of the procedure consists of ensuring no confounding or finding an adjustment set with directed information. In the second step, the causal effect is quantified. We subsequently unify previous definitions of directed information present in the literature and clarify the confusion surrounding them. We also motivate using chain graphs for directed information in time series and extend our approach to chain graphs. The proposed approach serves as a translation between causality modelling and information theory.

show abstract

Section: Proposed Methods For Causal Effect Identificationmentioning

confidence: 99%

“…Note that JSD is sometimes equivalently defined for

as symmetrised Kullback-Leibler divergence between

and

[ 77 ]. JSD has recently been applied in many machine learning areas such as GANs [ 78 ], bootstrapping [ 79 ], time series analysis [ 80 ] or computer vision [ 81 ].…”

Section: Proposed Methods For Causal Effect Identificationmentioning

confidence: 99%

Information Theoretic Causal Effect Quantification

Wieczorek

Röth

2019

Entropy

View full text Add to dashboard Cite

show abstract

“…Proof. (a) Since F and G are belonging to the same mixture family, JS(F, G) can be expressed as a Jensen-Bregman divergence [65]. Therefore, it can be written as:…”

Section: Conflicts Of Interestmentioning

confidence: 99%

Simplified Fréchet Distance for Generative Adversarial Nets

Kim

Jung

et al. 2020

Sensors

View full text Add to dashboard Cite

We introduce a distance metric between two distributions and propose a Generative Adversarial Network (GAN) model: the Simplified Fréchet distance (SFD) and the Simplified Fréchet GAN (SFGAN). Although the data generated through GANs are similar to real data, GAN often undergoes unstable training due to its adversarial structure. A possible solution to this problem is considering Fréchet distance (FD). However, FD is unfeasible to realize due to its covariance term. SFD overcomes the complexity so that it enables us to realize in networks. The structure of SFGAN is based on the Boundary Equilibrium GAN (BEGAN) while using SFD in loss functions. Experiments are conducted with several datasets, including CelebA and CIFAR-10. The losses and generated samples of SFGAN and BEGAN are compared with several distance metrics. The evidence of mode collapse and/or mode drop does not occur until 3000k steps for SFGAN, while it occurs between 457k and 968k steps for BEGAN. Experimental results show that SFD makes GANs more stable than other distance metrics used in GANs, and SFD compensates for the weakness of models based on BEGAN-based network structure. Based on the experimental results, we can conclude that SFD is more suitable for GAN than other metrics.

show abstract

“…Given this asymmetry, it is not a suitable candidate for a measure of covariate balance among groups: the divergence between two groups would depend upon which group is taken to be the Reference group. Jeffrey's divergence (J) is a symmetric version of relative entropy, defined as [7]. One reason why it is not a suitable candidate for the task of assessing covariate balance among groups is that there may be more than two groups.…”

Section: Relative Entropymentioning

confidence: 99%

An Information-Theoretic Measure for Balance Assessment in Comparative Clinical Studies

Dalton

Benish

Krieger

2020

Entropy

View full text Add to dashboard Cite

Limitations of statistics currently used to assess balance in observation samples include their insensitivity to shape discrepancies and their dependence upon sample size. The Jensen-Shannon divergence (JSD) is an alternative approach to quantifying the lack of balance among treatment groups that does not have these limitations. The JSD is an information-theoretic statistic derived from relative entropy, with three specific advantages relative to using standardized difference scores. First, it is applicable to cases in which the covariate is categorical or continuous. Second, it generalizes to studies in which there are more than two exposure or treatment groups. Third, it is decomposable, allowing for the identification of specific covariate values, treatment groups or combinations thereof that are responsible for any observed imbalance.

show abstract

On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means

Cited by 149 publications

References 56 publications

Information Theoretic Causal Effect Quantification

Information Theoretic Causal Effect Quantification

Simplified Fréchet Distance for Generative Adversarial Nets

An Information-Theoretic Measure for Balance Assessment in Comparative Clinical Studies

Contact Info

Product

Resources

About