Compressive Visual Representations

Abstract

Learning effective visual representations that generalize well without human supervision is a fundamental problem in order to apply Machine Learning to a wide variety of tasks. Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respectively, have made significant progress. In this work, we hypothesize that adding explicit information compression to these algorithms yields better and more robust representations. We verify this by developing SimCLR and BYOL formulations compatible with the Conditional Entropy Bottleneck (CEB) objective, allowing us to both measure and control the amount of compression in the learned representation, and observe their impact on downstream tasks. Furthermore, we explore the relationship between Lipschitz continuity and compression, showing a tractable lower bound on the Lipschitz constant of the encoders we learn. As Lipschitz continuity is closely related to robustness, this provides a new explanation for why compressed models are more robust. Our experiments confirm that adding compression to SimCLR and BYOL significantly improves linear evaluation accuracies and model robustness across a wide range of domain shifts. In particular, the compressed version of BYOL achieves 76.0% Top-1 linear evaluation accuracy on ImageNet with ResNet-50, and 78.8% with ResNet-50 2x.Recent contrastive approaches to self-supervised visual representation learning aim to learn representations that maximally capture the mutual information between two transformed views of an image [54,4,12,33,40]. The primary idea of these approaches is that this mutual information corresponds to a general shared context that is invariant to various transformations of the input, and it † Main contributors

Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?

“…2) in conjunction with a contrastive loss. (Chen et al, 2020b) 71.1 -InfoMin Aug. 73.0 91.1 BYOL 74.3 91.6 RELIC (Mitrovic et al, 2021) 74.8 92.2 SwAV (Caron et al, 2020) 75.3 -NNCLR (Dwibedi et al, 2021) 75.6 92.4 C-BYOL (Lee et al, 2021) 75 negative points with different strengths corresponding to the distance to the anchor point. These have the effect of separating the mean representation of the classes.…”

“…Next we evaluate the performance of RELICv2 in a semisupervised setting. We pretrain the representation and leverage a small subset of the available labels in the ImageNet training set to refine the learned representation following the protocol as described in (Chen et al, 2020a;Caron et al, 2020;Dwibedi et al, 2021;Lee et al, 2021) and the supplementary material. Top-1 and top-5 accuracy on the ImageNet validation set is reported in table 2.…”

“…Top-1 and top-5 accuracy on the ImageNet validation set is reported in table 2. RELICv2 outperforms both the supervised baseline and all previous state-of-the-art self-supervised methods when 53.2 68.8 78.4 89.00 SwAV (Caron et al, 2020) 53.9 70.2 78.5 89.9 NNCLR (Dwibedi et al, 2021) 56.4 69.8 80.7 89.3 C-BYOL (Lee et al, 2021) 60 using 10% of the data for fine-tuning. When using 1% of the data, only C-BYOL performs better than RELICv2.…”

“…A detailed comparison is provided in the supplementary material. Outside of this scope, (Lee et al, 2021) have taken a conditional entropy bottleneck approach which is particularly notable as it has also resulted in better-than-supervised performance albeit only on the ResNet50 (2x) architecture (see figure 1).…”

“…To train the linear classifier, we optimize the cross-entropy loss with stochastic gradient descent with Nestorov momentum for 100 epochs using a batch size of 1024 and a momentum of 0.9; we do not use any weight decay or other regularization techniques. In the following tables we report the top- (He et al, 2019) 65.4 -SimCLR (Chen et al, 2020a) 74.2 92.0 BYOL 77.4 93.6 SwAV (Caron et al, 2020) 77.3 -C-BYOL (Lee et al, 2021) 78.8 94.5 RELICv2 (ours) 79.0 94.5…”