Caveats for information bottleneck in deterministic scenarios

Kolchinsky, Artemy; Tracey, Brendan D.; Kuyk, Steven Van

doi:10.48550/arxiv.1808.07593

Cited by 9 publications

(12 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work analyzes deep neural networks through the lens of information theory [73][74][75][76][77][78][79][80][81][82][83], often computing measures of mutual information similar to those we discuss. Our result that the only usable information in a dataset is contained in its sample-sample second moment matrix K may inform or constrain this type of analysis.…”

Section: Discussionmentioning

confidence: 99%

Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization

Wadia¹,

Duckworth²,

Schoenholz³

et al. 2020

Preprint

View full text Add to dashboard Cite

Machine learning is predicated on the concept of generalization: a model achieving low error on a sufficiently large training set should also perform well on novel samples from the same distribution. We show that both data whitening and second order optimization can harm or entirely prevent generalization. In general, model training harnesses information contained in the sample-sample second moment matrix of a dataset. For a general class of models, namely models with a fully connected first layer, we prove that the information contained in this matrix is the only information which can be used to generalize. Models trained using whitened data, or with certain second order optimization schemes, have less access to this information; in the high dimensional regime they have no access at all, producing models that generalize poorly or not at all. We experimentally verify these predictions for several architectures, and further demonstrate that generalization continues to be harmed even when theoretical requirements are relaxed. However, we also show experimentally that regularized second order optimization can provide a practical tradeoff, where training is accelerated but less information is lost, and generalization can in some circumstances even improve. * Work done while an intern with Google Brain.Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization

Wadia¹,

Duckworth²,

Schoenholz³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The IB principle has recently been introduced for theoretical understanding and analysis of deep neural networks [2,21,34,43,48]. The authors optimize the networks with an iterative Blahut-Arimoto algorithm, which is infeasible in practical systems.…”

Section: Related Workmentioning

confidence: 99%

Learning to Learn with Variational Information Bottleneck for Domain Generalization

Xiong

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Domain generalization models learn to generalize to previously unseen domains, but suffer from prediction uncertainty and domain shift. In this paper, we address both problems. We introduce a probabilistic meta-learning model for domain generalization, in which classifier parameters shared across domains are modeled as distributions. This enables better handling of prediction uncertainty on unseen domains. To deal with domain shift, we learn domain-invariant representations by the proposed principle of meta variational information bottleneck, we call MetaVIB. MetaVIB is derived from novel variational bounds of mutual information, by leveraging the meta-learning setting of domain generalization. Through episodic training, MetaVIB learns to gradually narrow domain gaps to establish domain-invariant representations, while simultaneously maximizing prediction accuracy. We conduct experiments on three benchmarks for cross-domain visual recognition. Comprehensive ablation studies validate the benefits of MetaVIB for domain generalization. The comparison results demonstrate our method outperforms previous approaches consistently.

show abstract

“…= H(z) − H(z | y) where H(z) is the entropy of z [CT06]. 3 given one can overcome some caveats associated with this framework [KTVK18] and practical difficulties such as how to accurately evaluate mutual information with finitely samples of degenerate distributions. 4 in case the labels can be corrupted or the learned features be tackled.…”

Section: Context and Motivationmentioning

confidence: 99%

Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction

Chan

You

et al. 2020

Preprint

View full text Add to dashboard Cite

To learn intrinsic low-dimensional structures from high-dimensional data that most discriminate between classes, we propose the principle of Maximal Coding Rate Reduction (MCR 2 ), an information-theoretic measure that maximizes the coding rate difference between the whole dataset and the sum of each individual class. We clarify its relationships with most existing frameworks such as cross-entropy, information bottleneck, information gain, contractive and contrastive learning, and provide theoretical guarantees for learning diverse and discriminative features. The coding rate can be accurately computed from finite samples of degenerate subspace-like distributions and can learn intrinsic representations in supervised, self-supervised, and unsupervised settings in a unified manner. Empirically, the representations learned using this principle alone are significantly more robust to label corruptions in classification than those using cross-entropy, and can lead to state-of-the-art results in clustering mixed data from self-learned invariant features.

show abstract

Caveats for information bottleneck in deterministic scenarios

Cited by 9 publications

References 23 publications

Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization

Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization

Learning to Learn with Variational Information Bottleneck for Domain Generalization

Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction

Contact Info

Product

Resources

About