Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss

Abstract

Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm, which learns representations by pushing positive pairs, or similar examples from the same class, closer together while keeping negative pairs far apart. Despite the empirical successes, theoretical foundations are limited -prior analyses assume conditional independence of the positive pairs given the same class label, but recent empirical applications use heavily correlated positive pairs (i.e., data augmentations of the same image). Our work analyzes contrastive learning without assuming conditional independence of positive pairs using a novel concept of the augmentation graph on data. Edges in this graph connect augmentations of the same data, and ground-truth classes naturally form connected sub-graphs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective on neural net representations. Minimizing this objective leads to features with provable accuracy guarantees under linear probe evaluation. By standard generalization bounds, these accuracy guarantees also hold when minimizing the training contrastive loss. Empirically, the features learned by our objective can match or outperform several strong baselines on benchmark vision datasets. In all, this work provides the first provable analysis for contrastive learning where guarantees for linear probe evaluation can apply to realistic empirical settings.

Investigating Why Contrastive Learning Benefits Robustness Against Label Noise

“…In particular, it is shown that under conditional independence given the label and/or additional latent variables, representations learned by reconstruction-based self-supervised learning algorithms can achieve small errors in the downstream linear classification task (Arora et al, 2019;Tosh et al, 2021). More closely related to our work is the recent result of HaoChen et al (2021) that analyzed contrastive learning without assuming conditional independence of positive pairs. Based on the concept of augmentation graph, they showed that spectral decomposition on the augmented distribution leads to embeddings with provable accuracy guarantees under linear probe evaluation.…”

“…Theoretical works on self-supervised learning. A recent line of theoretical works have studied selfsupervised learning (Arora et al, 2019;Tosh et al, 2021;HaoChen et al, 2021). In particular, it is shown that under conditional independence given the label and/or additional latent variables, representations learned by reconstruction-based self-supervised learning algorithms can achieve small errors in the downstream linear classification task (Arora et al, 2019;Tosh et al, 2021).…”

“…Augmentation graph. The augmentations can be used to construct the population augmentation graph (HaoChen et al, 2021), whose vertices are all the augmented data in the population distribution, and two vertices are connected with an edge if they are augmentations of the same natural example. Hence, ground-truth classes naturally form connected sub-graphs.…”

“…The embeddings produced by contrastive learning can be viewed as a low-rank approximation of the normalized augmentation graph. Effectively, minimizing a loss that performs spectral decomposition on the population augmentation graph can be succinctly written as a contrastive learning objective C(f ) on neural network representations (HaoChen et al, 2021):…”

“…An important technical idea we use to formalize the representations obtained by contrastive learning is augmentation graph (HaoChen et al, 2021), which is an undirected graph with all augmentation data {x 1 , x 2 , . .…”