Mean Shift for Self-Supervised Learning

Koohpayegani, Soroush Abbasi; Tejankar, Ajinkya; Pirsiavash, Hamed

doi:10.1109/iccv48922.2021.01016

Cited by 64 publications

(24 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, since the combined patch embeddings only contain part of the information in the whole image, pulling the partially combined patches closer to the target view that contains the whole image information is more challenging than pulling the original image pairs and implicitly increasing the asymmetric of the network structure, which have been demonstrated beneficial for increasing the richness of feature representations and improve the self-supervised learning performance [15,11,22]. Owing to these merits, Fast-MoCo can achieve high sample utilization efficiency with marginal extra computational cost and thus obtain promising performance with much less training time.…”

Section: Discussionmentioning

confidence: 99%

Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches

Ci¹,

Chen²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

Contrastive-based self-supervised learning methods achieved great success in recent years. However, self-supervision requires extremely long training epochs (e.g., 800 epochs for MoCo v3) to achieve promising results, which is unacceptable for the general academic community and hinders the development of this topic. This work revisits the momentumbased contrastive learning frameworks and identifies the inefficiency in which two augmented views generate only one positive pair. We propose Fast-MoCo -a novel framework that utilizes combinatorial patches to construct multiple positive pairs from two augmented views, which provides abundant supervision signals that bring significant acceleration with neglectable extra computational cost. Fast-MoCo trained with 100 epochs achieves 73.5% linear evaluation accuracy, similar to MoCo v3 (ResNet-50 backbone) trained with 800 epochs. Extra training (200 epochs) further improves the result to 75.1%, which is on par with stateof-the-art methods. Experiments on several downstream tasks also confirm the effectiveness of Fast-MoCo.

show abstract

Section: Discussionmentioning

confidence: 99%

Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches

Ci¹,

Chen²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…NNCLR [23] compares with SimCLR to select the most similar sample representation from a queue by the nearest neighbor method instead of the original representation to calculate the contrastive loss, thus improving the model performance by increasing the training complexity. MSF [24] compared to SimCLR calculates the contrastive loss by selecting the K most similar sample representations in a queue and calculating the mean instead of the original sample representation. TTL [25] and HardCL [26] sensor data encoder encoder However, with the downstream task identified as a classification task, the above work is contradictory for how negative examples are defined because this definition would consider each instance as a single class.…”

Section: Related Work a Definition Of Contrastive Learning Negative E...mentioning

confidence: 99%

“…The pre-trained learned model is fine-tuned in downstream tasks using a small amount of labeled data to achieve performance comparable to supervised learning [16] [17]. There are many types of pre-training tasks for contrastive learning, such as MoCo [18] [19] and SimCLR [20] [21] with instance discrimination [22] as the task, and NNCLR [23], MSF [24], TTL [25] and HardCL [26] which redefine positive and negative pairs based on the instance discrimination task. In addition to this, SwAV [27] uses clustering to reduce feature dimensionality, BYOL [28] and SimSiam [29] that drop the use of negative examples and use similarity metrics for the pretraining task.…”

Section: Introductionmentioning

confidence: 99%

Negative Selection by Clustering for Contrastive Learning in Human Activity Recognition

Wang¹,

Zhu²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Contrastive learning has been applied to Human Activity Recognition (HAR) based on sensor data owing to its ability to achieve performance comparable to supervised learning with a large amount of unlabeled data and a small amount of labeled data. The pre-training task for contrastive learning is generally instance discrimination, which specifies that each instance belongs to a single class, but this will consider the same class of samples as negative examples. Such a pretraining task is not conducive to human activity recognition tasks, which are mainly classification tasks. To address this problem, we follow SimCLR to propose a new contrastive learning framework that negative selection by clustering in HAR, which is called ClusterCLHAR. Compared with SimCLR, it redefines the negative pairs in the contrastive loss function by using unsupervised clustering methods to generate soft labels that mask other samples of the same cluster to avoid regarding them as negative samples. We evaluate ClusterCLHAR on three benchmark datasets, USC-HAD, MotionSense, and UCI-HAR, using mean F1-score as the evaluation metric. The experiment results show that it outperforms all the state-of-the-art methods applied to HAR in self-supervised learning and semi-supervised learning.

show abstract

“…For example, pictures of different cats should not be considered completely negative to each other. To this end, heuristically modifying the inter-sample relations have been widely applied in selfsupervised learning [10,12,22,34], intuitively similar to us. Many of the state-of-the-art methods can be considered as special cases of our method by the adjustment of hyperparameters (Sec.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Soft Contrastive Learning

Feng

Patras

2022

2022 26th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Deep learning has achieved great success in recent years with the aid of advanced neural network structures and large-scale human-annotated datasets. However, it is often costly and difficult to accurately and efficiently annotate large-scale datasets, especially for some specialized domains where fine-grained labels are required. In this setting, coarse labels are much easier to acquire as they do not require expert knowledge. In this work, we propose a contrastive learning method, called masked contrastive learning (MaskCon) to address the under-explored problem setting, where we learn with a coarse-labelled dataset in order to address a finer labelling problem. More specifically, within the contrastive learning framework, for each sample our method generates soft-labels with the aid of coarse labels against other samples and another augmented view of the sample in question. By contrast to self-supervised contrastive learning where only the sample's augmentations are considered hard positives, and in supervised contrastive learning where only samples with the same coarse labels are considered hard positives, we propose soft labels based on sample distances, that are masked by the coarse labels. This allows us to utilize both inter-sample relations and coarse labels. We demonstrate that our method can obtain as special cases many existing state-of-the-art works and that it provides tighter bounds on the generalization error. Experimentally, our method achieves significant improvement over the current state-of-the-art in various datasets, including CIFAR10, CIFAR100, ImageNet-1K, Standford Online Products and Stanford Cars196 datasets. Code and annotations are available at https://github.com/ MrChenFeng/MaskCon_CVPR2023.

show abstract

Mean Shift for Self-Supervised Learning

Abstract: County (UMBC)ScholarWorks@UMBC digital repository on the Maryland Shared Open Access (MD-SOAR) platform.

Cited by 64 publications

References 33 publications

Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches

Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches

Negative Selection by Clustering for Contrastive Learning in Human Activity Recognition

Adaptive Soft Contrastive Learning

Contact Info

Product

Resources

About