On the duality between contrastive and non-contrastive self-supervised learning

Abstract

Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we focus more on the theoretical similarities between them. By designing contrastive and non-contrastive criteria that can be related algebraically and shown to be equivalent under limited assumptions, we show how close those families can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show how design choices in the criterion can influence the optimization process and downstream performance. We also challenge the popular assumptions that contrastive and non-contrastive methods, respectively, need large batch sizes and output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and noncontrastive methods in certain regimes can be significantly reduced given better network design choice and hyperparameter tuning.

GTOC 11: Results from Tsinghua University and Shanghai Institute of Satellite Engineering

“…Various theoretical studies have also investigated non-contrastive methods for self-supervised learning [71,61,18,5,66,49,57,34]. Garrido et al [18] establishes the duality between contrastive and non-contrastive methods.…”

“…Various theoretical studies have also investigated non-contrastive methods for self-supervised learning [71,61,18,5,66,49,57,34]. Garrido et al [18] establishes the duality between contrastive and non-contrastive methods. Balestriero & LeCun [5] reveals the connections between variants of SimCLR, Barlow Twins, and VICReg to ISOMAP, Canonical Correlation Analysis, and Laplacian Eigenmaps, respectively.…”

“…Many theoretical studies have also paid attention to the intrinsic mechanism behind non-contrastive SSL methods. For example, Garrido et al [18] established the duality between contrastive and non-contrastive methods, and Balestriero & LeCun [5] unveiled the links of SimCLR, Barlow Twins and VICReg to spectral methods.…”

Intra-Instance VICReg: Bag of Self-Supervised Image Patch Embedding

“…A thorough discussion is beyond the scope of this work. We refer the curious readers to Garrido et al [2022] for a more general discussion on the duality between contrastive learning and non-contrastive learning.…”

DyG2Vec: Representation Learning for Dynamic Graphs with Self-Supervision

“…However, it is important to specify a pre-training objective function that induces good performance for the downstream tasks. Contrastive SSL methods, despite their early success, rely heavily on negative samples, extensive data augmentation, and large batch sizes (Jing et al, 2022;Garrido et al, 2022). Non-contrastive methods address these shortcomings, incorporating information theoretic principles through architectural innovations or regularization methods.…”