Conditional Mutual Information Neural Estimator

Molavipour, Sina; Bassi, Germán; Skoglund, Mikael

doi:10.1109/icassp40776.2020.9053422

Cited by 13 publications

(12 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3 and 4. Here, we have chosen α = α * in (46) so that the value of the critical coupling K c matches the exact result given in (44). As expected from our analysis above, we observe a qualitative difference in behavior for K above vs. below the critical value K c .…”

Section: D Classical Ising Modelsupporting

confidence: 60%

“…The modification due to Maris and Kadanoff in (41) corresponds to α = 3 2 , which gives a non-trivial fixed point at (42). The value α * of α that would produce the exact critical coupling (44) is slightly greater than this:…”

Section: D Classical Ising Modelmentioning

confidence: 98%

“…Mutual information in particular -which may be defined as the KL divergence of p(x)p( y) relative to p(x, y) -has a wide utility ranging from the information bottleneck [37] to various maximization (training) methods [38][39][40][41][42]. For a non-exhaustive sampling of other works on estimating or bounding entropic quantities in deep learning, see [43][44][45][46][47].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

Erdmenger

Jefferson

2022

SciPost Phys.

View full text Add to dashboard Cite

We investigate the analogy between the renormalization group (RG) and deep neural networks, wherein subsequent layers of neurons are analogous to successive steps along the RG. In particular, we quantify the flow of information by explicitly computing the relative entropy or Kullback-Leibler divergence in both the one- and two-dimensional Ising models under decimation RG, as well as in a feedforward neural network as a function of depth. We observe qualitatively identical behavior characterized by the monotonic increase to a parameter-dependent asymptotic value. On the quantum field theory side, the monotonic increase confirms the connection between the relative entropy and the c-theorem. For the neural networks, the asymptotic behavior may have implications for various information maximization methods in machine learning, as well as for disentangling compactness and generalizability. Furthermore, while both the two-dimensional Ising model and the random neural networks we consider exhibit non-trivial critical points, the relative entropy appears insensitive to the phase structure of either system. In this sense, more refined probes are required in order to fully elucidate the flow of information in these models.

show abstract

Section: D Classical Ising Modelsupporting

confidence: 60%

Section: D Classical Ising Modelmentioning

confidence: 98%

mentioning

confidence: 99%

See 1 more Smart Citation

Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

Erdmenger

Jefferson

2022

SciPost Phys.

View full text Add to dashboard Cite

show abstract

“…Several variants of these bounds have been reviewed in [ 9 ]. Variational bounds are tight, and the estimators proposed in [ 1 , 4 , 10 , 11 ] leverage this property and use neural networks to approximate the bounds and correspondingly the desired information measure. These estimators were shown to be consistent (i.e., the estimation converges asymptotically to the true value) and suitably estimate MI and CMI when the samples are independently and identically distributed (i.i.d.).…”

Section: Introductionmentioning

confidence: 99%

Neural Estimator of Information for Time-Series Data with Dependency

Molavipour

Ghourchian

Bassi

et al. 2021

Entropy

Self Cite

View full text Add to dashboard Cite

Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. {In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff's ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations.

show abstract

“…However, these mutual information approximations are lower bounds and cannot be used as approximations for the leakage as an upper bound would be needed. A possible workaround for certain channel is given by [13] which shows how a conditional mutual information can be estimated. iii) Reinforcement learning (RL):…”

Section: Introductionmentioning

confidence: 99%

Reinforce Security: A Model-Free Approach Towards Secure Wiretap Coding

Fritschek¹,

Schaefer²,

Wunder³

2021

Preprint

View full text Add to dashboard Cite

The use of deep learning-based techniques for approximating secure encoding functions has attracted considerable interest in wireless communications due to impressive results obtained for general coding and decoding tasks for wireless communication systems. Of particular importance is the development of model-free techniques that work without knowledge about the underlying channel. Such techniques utilize for example generative adversarial networks to estimate and model the conditional channel distribution, mutual information estimation as a reward function, or reinforcement learning. In this paper, the approach of reinforcement learning is studied and, in particular, the policy gradient method for a model-free approach of neural network-based secure encoding is investigated. Previously developed techniques for enforcing a certain coset structure on the encoding process can be combined with recent reinforcement learning approaches. This new approach is evaluated by extensive simulations, and it is demonstrated that the resulting decoding performance of an eavesdropper is capped at a certain error level.

show abstract

Conditional Mutual Information Neural Estimator

Cited by 13 publications

References 28 publications

Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

Neural Estimator of Information for Time-Series Data with Dependency

Reinforce Security: A Model-Free Approach Towards Secure Wiretap Coding

Contact Info

Product

Resources

About