Contrasting the landscape of contrastive and non-contrastive learning

Ashwini, Pokle,; Tian, Jinjin; Li, Yuchen; Risteski, Andrej

doi:10.48550/arxiv.2203.15702

Cited by 4 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various theoretical studies have also investigated non-contrastive methods for self-supervised learning [71,61,18,5,66,49,57,34]. Garrido et al [18] establishes the duality between contrastive and non-contrastive methods.…”

Section: Related Workmentioning

confidence: 99%

GTOC 11: Results from Tsinghua University and Shanghai Institute of Satellite Engineering

Zhang

Xiang

et al. 2023

Acta Astronautica

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

GTOC 11: Results from Tsinghua University and Shanghai Institute of Satellite Engineering

Zhang

Xiang

et al. 2023

Acta Astronautica

View full text Add to dashboard Cite

“…The fact that a method like SimSiam does not collapse is studied in [29]. The loss landscape of SimSiam is also compared to SimCLR's in [26], which shows that it learns bad minima. In [31], the optimal solutions of the InfoNCE criterion are characterized, giving a better understanding of the embedding distributions.…”

Section: Related Workmentioning

confidence: 99%

On the duality between contrastive and non-contrastive self-supervised learning

Garrido,

Chen,

Bardes

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we focus more on the theoretical similarities between them. By designing contrastive and non-contrastive criteria that can be related algebraically and shown to be equivalent under limited assumptions, we show how close those families can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show how design choices in the criterion can influence the optimization process and downstream performance. We also challenge the popular assumptions that contrastive and non-contrastive methods, respectively, need large batch sizes and output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and noncontrastive methods in certain regimes can be significantly reduced given better network design choice and hyperparameter tuning.

show abstract

“…Especially the theoretical papers by Tian et al [82] and Wang et al [87]. Pokle et al [67] compared the landscapes between contrastive and non-contrastive learning and points out the existence of non-collapsed bad minima for non-contrastive learning without a prediction head.…”

Section: Comparison To Similar Studiesmentioning

confidence: 99%

“…There are already some theoretical papers [82,87,67] that try to address similar questions. While none of these papers studied the training process of the prediction head, our results provide a completely different perspective: We explain why training the prediction head can encourage the network to learn diversified features and avoid dimensional collapses, even when the trivial collapsed optima still exist in the training objective, which is not covered by the prior works.…”

Section: Introductionmentioning

confidence: 99%

The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning

Wen¹,

Li²

2022

Preprint

View full text Add to dashboard Cite

Recently the surprising discovery of Bootstrap Your Own Latent (BYOL) method by Grill et al.shows the negative term in contrastive loss can be removed if we add the so-called prediction head to the network architecture, which breaks the symmetry between the positive pairs. This initiated the research of non-contrastive self-supervised learning. It is mysterious why even when trivial collapsed global optimal solutions exist, neural networks trained by (stochastic) gradient descent can still learn competitive representations and avoid collapsed solutions. This phenomenon is one of the most typical examples of implicit bias in deep learning optimization, and its underlying mechanism remains little understood to this day.In this work, we present our empirical and theoretical discoveries about the mechanism of prediction head in non-contrastive self-supervised learning methods. Empirically, we find that when the prediction head is initialized as an identity matrix with only its off-diagonal entries being trained, the network can learn competitive representations even though the trivial optima still exist in the training objective. Moreover, we observe a consistent rise and fall trajectory of off-diagonal entries during training. Our evidence suggests that understanding the identity-initialized prediction head is a good starting point for understanding the mechanism of the trainable prediction head.Theoretically, we present a framework to understand the behavior of the trainable, but identity-initialized prediction head. Under a simple setting, we characterized the substitution effect and acceleration effect of the prediction head during the training process. The substitution effect happens when learning the stronger features in some neurons can substitute for learning these features in other neurons through updating the prediction head. And the acceleration effect happens when the substituted features can accelerate the learning of other weaker features to prevent them from being ignored. These two effects together enable the neural networks to learn all the features rather than focus only on learning the stronger features, which is likely the cause of the dimensional collapse phenomenon. To the best of our knowledge, this is also the first end-to-end optimization guarantee for non-contrastive methods using nonlinear neural networks with a trainable prediction head and normalization.

show abstract

Contrasting the landscape of contrastive and non-contrastive learning

Cited by 4 publications

References 11 publications

GTOC 11: Results from Tsinghua University and Shanghai Institute of Satellite Engineering

GTOC 11: Results from Tsinghua University and Shanghai Institute of Satellite Engineering

On the duality between contrastive and non-contrastive self-supervised learning

The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning

Contact Info

Product

Resources

About