Direct Validation of the Information Bottleneck Principle for Deep Nets

Elad, A.; Haviv, Doron; Blau, Yochai; Michaeli, Tomer

doi:10.1109/iccvw.2019.00099

Cited by 25 publications

(25 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another type of variational lower bound can be obtained from the Donsker-Varadhan representation of mutual information, which is the principle underlying MINE [34]. MINE was used to estimate the IPs in [12] and [15] for a 784 − 512 − 512 − 10 MLP with ReLU or tanh activation functions trained on MNIST and a VGG-16 CNN trained on CIFAR-10, respectively. Reference [11] used a NN classifier and a generative PixelCNN++ [35] as variational distributions for analysing the IP of a ResNet trained on CINIC-10.…”

Section: Variational and Neural Network-based Estimatorsmentioning

confidence: 99%

On Information Plane Analyses of Neural Network Classifiers -- A Review

Geiger¹

2020

Preprint

View full text Add to dashboard Cite

We review the current literature concerned with information plane analyses of neural network classifiers. While the underlying information bottleneck theory and the claim that information-theoretic compression is causally linked to generalization are plausible, empirical evidence was found to be both supporting and conflicting. We review this evidence together with a detailed analysis how the respective information quantities were estimated. Our analysis suggests that compression visualized in information planes is not information-theoretic, but is rather compatible with geometric compression of the activations.

show abstract

Section: Variational and Neural Network-based Estimatorsmentioning

confidence: 99%

On Information Plane Analyses of Neural Network Classifiers -- A Review

Geiger¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, the IB principle has been proposed for analyzing and understanding the dynamics of learning and the generalization of DNNs [14]. [15] applies the recently proposed mutual information neural estimator (MINE) [16] to train hidden layer with IB loss, and freeze it before moving on to the next layer. Although the result corroborates partially the IB hypothesis in DNNs, some claims are still controversial [17,18].…”

Section: Ib Principle and Deep Neural Networkmentioning

confidence: 99%

Deep Deterministic Information Bottleneck with Matrix-Based Entropy Functional

Yu²,

Prı́ncipe

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We introduce the matrix-based Rényi's α-order entropy functional to parameterize Tishby et al. information bottleneck (IB) principle [1] with a neural network. We term our methodology Deep Deterministic Information Bottleneck (DIB), as it avoids variational inference and distribution assumption. We show that deep neural networks trained with DIB outperform the variational objective counterpart and those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack. Code available at https://github.com/yuxi120407/DIB.

show abstract

“…Pairs {(x (i) , z (i) )} come from joint distribution p(x, z), while samples {ž (i) } come from marginal distribution p(z). It has been used for analyzing mutual information between layers of neural networks [32,33].…”

Section: Non-parametric Estimation Of Mutual Informationmentioning

confidence: 99%

Information Bottleneck Analysis by a Conditional Mutual Information Bound

Tezuka

Namekawa

2021

Entropy

View full text Add to dashboard Cite

Task-nuisance decomposition describes why the information bottleneck loss I(z;x)−βI(z;y) is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, I(z;n) can be decreased by reducing I(z;x) since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information I(z;x|y) provides an alternative upper bound for I(z;n). This bound is applicable even if z is not a sufficient representation of x, that is, I(z;y)≠I(x;y). We used mutual information neural estimation (MINE) to estimate I(z;x|y). Experiments demonstrated that I(z;x|y) is smaller than I(z;x) for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when I(z;x|y) is used instead of I(z;x).

show abstract

Direct Validation of the Information Bottleneck Principle for Deep Nets

Cited by 25 publications

References 5 publications

On Information Plane Analyses of Neural Network Classifiers -- A Review

On Information Plane Analyses of Neural Network Classifiers -- A Review

Deep Deterministic Information Bottleneck with Matrix-Based Entropy Functional

Information Bottleneck Analysis by a Conditional Mutual Information Bound

Contact Info

Product

Resources

About