Contrastive learning, multi-view redundancy, and linear models

Tosh, Christopher; Krishnamurthy, Akshay; Hsu, Daniel

doi:10.48550/arxiv.2008.10150

Cited by 10 publications

(12 citation statements)

References 20 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Class Collision and Uniformity for One-Class Classification. While contrastive representations have achieved state-of-the-art performance on visual recognition tasks (Oord et al, 2018;Hénaff et al, 2019;Wang & Isola, 2020) and have been theoretically proved to be effective for multi-class classification (Saunshi et al, 2019;Tosh et al, 2020), we argue that this could be problematic for one-class classification.…”

Section: Contrastive Learningmentioning

confidence: 70%

Learning and Evaluating Representations for Deep One-class Classification

Sohn,

Li,

Yoon

et al. 2020

Preprint

View full text Add to dashboard Cite

We present a two-stage framework for deep one-class classification. We first learn self-supervised representations from one-class data, and then build one-class classifiers on learned representations. The framework not only allows to learn better representations, but also permits building one-class classifiers that are faithful to the target task. In particular, we present a novel distribution-augmented contrastive learning that extends training distributions via data augmentation to obstruct the uniformity of contrastive representations. Moreover, we argue that classifiers inspired by the statistical perspective in generative or discriminative models are more effective than existing approaches, such as an average of normality scores from a surrogate classifier. In experiments, we demonstrate state-of-the-art performance on visual domain one-class classification benchmarks. Finally, we present visual explanations, confirming that the decision-making process of our deep one-class classifier is intuitive to humans. The code is available at: https://github.com/google-research/google-research/ tree/master/deep_representation_one_class.

show abstract

Section: Contrastive Learningmentioning

confidence: 70%

Learning and Evaluating Representations for Deep One-class Classification

Sohn,

Li,

Yoon

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…While many theoretical results on contrastive SSL (Arora et al, 2019;Lee et al, 2020;Tosh et al, 2020;Wen & Li, 2021) do exist, similar study on nc-SSL has been very rare. As one of the first work towards this direction, Tian et al (2021) show that while the global optimum of the non-contrastive loss is indeed a trivial one, following gradient direction in nc-SSL, one can find a local optimum that admits a nontrivial representation.…”

Section: Introductionmentioning

confidence: 99%

Towards Demystifying Representation Learning with Non-contrastive Self-supervision

Wang¹,

Chen²,

Du³

et al. 2021

Preprint

View full text Add to dashboard Cite

Non-contrastive methods of self-supervised learning (such as BYOL and SimSiam) learn representations by minimizing the distance between two views of the same image. These approaches have achieved remarkable performance in practice, but it is not well understood 1) why these methods do not collapse to the trivial solutions and 2) how the representation is learned. Tian et al. ( 2021) made an initial attempt on the first question and proposed DirectPred that sets the predictor directly. In our work, we analyze a generalized version of DirectPred, called DirectSet(α). We show that in a simple linear network, DirectSet(α) provably learns a desirable projection matrix and also reduces the sample complexity on downstream tasks. Our analysis suggests that weight decay acts as an implicit threshold that discard the features with high variance under augmentation, and keep the features with low variance. Inspired by our theory, we simplify DirectPred by removing the expensive eigen-decomposition step. On CIFAR-10, CIFAR-100, STL-10 and ImageNet, DirectCopy, our simpler and more computationally efficient algorithm, rivals or even outperforms DirectPred.

show abstract

“…In this paper, we further study the situation that the CI condition does not hold, and explore the idea of applying a learnable function to the input to make the CI condition hold. Other works (Saunshi et al, 2019;Tosh et al, 2020) study the generalization error of contrastive learning based SSL, whose setting is different from our paper. Most recently, Bansal et al (2020) analyze the generalization gap for most SSL methods.…”

Section: Related Workmentioning

confidence: 71%

Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis

Teng¹,

Huang²,

He³

2021

Preprint

View full text Add to dashboard Cite

Pretext-based self-supervised learning aims to learn the semantic representation via a handcrafted pretext task over unlabeled data and then use the learned representation for downstream prediction tasks. Lee et al. (2020) prove that pretextbased self-supervised learning can effectively reduce the sample complexity of downstream tasks under Conditional Independence (CI) between the components of the pretext task conditional on the downstream label. However, the CI condition rarely holds in practice, and the downstream sample complexity will get much worse if the CI condition does not hold. In this paper, we explore the idea of applying a learnable function to the input to make the CI condition hold. In particular, we first rigorously formulate the criteria that the function needs to satisfy. We then design an ingenious loss function for learning such a function and prove that the function minimizing the proposed loss satisfies the above criteria. We theoretically study the number of labeled data required, and give a model-free lower bound showing that taking limited downstream data will hurt the performance of self-supervised learning. Furthermore, we take the model structure into account and give a model-dependent lower bound, which gets higher when the model capacity gets larger. Moreover, we conduct several numerical experiments to verify our theoretical results.

show abstract

Contrastive learning, multi-view redundancy, and linear models

Cited by 10 publications

References 20 publications

Learning and Evaluating Representations for Deep One-class Classification

Learning and Evaluating Representations for Deep One-class Classification

Towards Demystifying Representation Learning with Non-contrastive Self-supervision

Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis

Contact Info

Product

Resources

About