Lossy Compression for Lossless Prediction

Dubois, Yann; Bloem-Reddy, Benjamin; Ullrich, Karen; Maddison, Chris J.

doi:10.48550/arxiv.2106.10800

Cited by 3 publications

(9 citation statements)

References 77 publications

(123 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…which recovers the optimal encoder in the case of unconstrained variational families for p ϕ , q θ , s ψ , infinite samples n → ∞, and any λ > 1 (Dubois et al, 2021).…”

Section: B4 Augmentationsmentioning

confidence: 84%

“…E.g, in CLIP, images are augmented with alt-text collected on the internet and invariance is enforced between the representations of the image and its text pair (Radford et al, 2021). Representations learned like this preserve discriminative information about all downstream tasks Y whose label information is preserved by the augmentation (e.g., Dubois et al, 2021).…”

Section: Self-supervised Learning Using Domain-agnostic Augmentationsmentioning

confidence: 99%

“…An important result that we will be using is the generalized data processing inequality of Bayes risk (Xu & Raginsky, 2020;Dubois et al, 2021). We include it here for completeness.…”

Section: B Proofs B1 Lemmas For General Lossesmentioning

confidence: 99%

“…Lemma 1 (Generalized DPI (Xu & Raginsky, 2020;Dubois et al, 2021)). Let Z − X − Y be a Markov chain of random variables.…”

Section: B Proofs B1 Lemmas For General Lossesmentioning

confidence: 99%

“…By Assumption 10, f * is invariant on the equivalence classes [x] := {x ∈ X | x ∼ x} for all x ∈ X . Thus, there exists a function g : N → A such that f * = g • M (Lemma 5, Dubois et al, 2021). Given z ∈ supp(p Z ), we construct h * in the following way.…”

Section: B4 Augmentationsmentioning

confidence: 99%

See 4 more Smart Citations

Optimal Representations for Covariate Shift

Ruan¹,

Dubois²,

Maddison³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning systems often experience a distribution shift between training and testing. In this paper, we introduce a simple variational objective whose optima are exactly the set of all representations on which risk minimizers are guaranteed to be robust to any distribution shift that preserves the Bayes predictor, e.g., covariate shifts. Our objective has two components. First, a representation must remain discriminative for the task, i.e., some predictor must be able to simultaneously minimize the source and target risk. Second, the representation's marginal support needs to be the same across source and target. We make this practical by designing self-supervised learning methods that only use unlabelled data and augmentations to train robust representations. Our objectives achieve state-of-the-art results on DomainBed, and give insights into the robustness of recent methods, such as CLIP.

show abstract

“…which recovers the optimal encoder in the case of unconstrained variational families for p ϕ , q θ , s ψ , infinite samples n → ∞, and any λ > 1 (Dubois et al, 2021).…”

Section: B4 Augmentationsmentioning

confidence: 84%

Section: Self-supervised Learning Using Domain-agnostic Augmentationsmentioning

confidence: 99%

“…An important result that we will be using is the generalized data processing inequality of Bayes risk (Xu & Raginsky, 2020;Dubois et al, 2021). We include it here for completeness.…”

Section: B Proofs B1 Lemmas For General Lossesmentioning

confidence: 99%

“…Lemma 1 (Generalized DPI (Xu & Raginsky, 2020;Dubois et al, 2021)). Let Z − X − Y be a Markov chain of random variables.…”

Section: B Proofs B1 Lemmas For General Lossesmentioning

confidence: 99%

Section: B4 Augmentationsmentioning

confidence: 99%

See 3 more Smart Citations

Optimal Representations for Covariate Shift

Ruan¹,

Dubois²,

Maddison³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Compressive Visual Representations

Lee¹,

Arnab²,

Guadarrama³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning effective visual representations that generalize well without human supervision is a fundamental problem in order to apply Machine Learning to a wide variety of tasks. Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respectively, have made significant progress. In this work, we hypothesize that adding explicit information compression to these algorithms yields better and more robust representations. We verify this by developing SimCLR and BYOL formulations compatible with the Conditional Entropy Bottleneck (CEB) objective, allowing us to both measure and control the amount of compression in the learned representation, and observe their impact on downstream tasks. Furthermore, we explore the relationship between Lipschitz continuity and compression, showing a tractable lower bound on the Lipschitz constant of the encoders we learn. As Lipschitz continuity is closely related to robustness, this provides a new explanation for why compressed models are more robust. Our experiments confirm that adding compression to SimCLR and BYOL significantly improves linear evaluation accuracies and model robustness across a wide range of domain shifts. In particular, the compressed version of BYOL achieves 76.0% Top-1 linear evaluation accuracy on ImageNet with ResNet-50, and 78.8% with ResNet-50 2x.Recent contrastive approaches to self-supervised visual representation learning aim to learn representations that maximally capture the mutual information between two transformed views of an image [54,4,12,33,40]. The primary idea of these approaches is that this mutual information corresponds to a general shared context that is invariant to various transformations of the input, and it † Main contributors

show abstract

Information-theoretic Online Memory Selection for Continual Learning

Sun¹,

Calandriello²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

A challenging problem in task-free continual learning is the online selection of a representative replay memory from data streams. In this work, we investigate the online memory selection problem from an information-theoretic perspective. To gather the most information, we propose the surprise and the learnability criteria to pick informative points and to avoid outliers. We present a Bayesian model to compute the criteria efficiently by exploiting rank-one matrix structures. We demonstrate that these criteria encourage selecting informative points in a greedy algorithm for online memory selection. Furthermore, by identifying the importance of the timing to update the memory, we introduce a stochastic informationtheoretic reservoir sampler (InfoRS), which conducts sampling among selective points with high information. Compared to reservoir sampling, InfoRS demonstrates improved robustness against data imbalance. Finally, empirical performances over continual learning benchmarks manifest its efficiency and efficacy.

show abstract

Lossy Compression for Lossless Prediction

Cited by 3 publications

References 77 publications

Optimal Representations for Covariate Shift

Optimal Representations for Covariate Shift

Compressive Visual Representations

Information-theoretic Online Memory Selection for Continual Learning

Contact Info

Product

Resources

About