ScatSimCLR: self-supervised contrastive learning with pretext task regularization for small-scale datasets

Kinakh, Vitaliy; Taran, Olga; Voloshynovskiy, Slava

doi:10.1109/iccvw54120.2021.00129

Cited by 8 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ADC [48] 53.0 IIC [49] 61 TSUK [50] 66.5 SCAN [16] 80.9 ScatSimCLR [30] 85.1 RUC [51] 86.7 MV-MR (ours) 89.7…”

Section: Methods Stl10mentioning

confidence: 99%

See 1 more Smart Citation

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

Kinakh,

Drozdova,

Voloshynovskiy

2024

Entropy

Self Cite

View full text Add to dashboard Cite

We present a new method of self-supervised learning and knowledge distillation based on multi-views and multi-representations (MV–MR). MV–MR is based on the maximization of dependence between learnable embeddings from augmented and non-augmented views, jointly with the maximization of dependence between learnable embeddings from the augmented view and multiple non-learnable representations from the non-augmented view. We show that the proposed method can be used for efficient self-supervised classification and model-agnostic knowledge distillation. Unlike other self-supervised techniques, our approach does not use any contrastive learning, clustering, or stop gradients. MV–MR is a generic framework allowing the incorporation of constraints on the learnable embeddings via the usage of image multi-representations as regularizers. The proposed method is used for knowledge distillation. MV–MR provides state-of-the-art self-supervised performance on the STL10 and CIFAR20 datasets in a linear evaluation setup. We show that a low-complexity ResNet50 model pretrained using proposed knowledge distillation based on the CLIP ViT model achieves state-of-the-art performance on STL10 and CIFAR100 datasets.

show abstract

“…ADC [48] 53.0 IIC [49] 61 TSUK [50] 66.5 SCAN [16] 80.9 ScatSimCLR [30] 85.1 RUC [51] 86.7 MV-MR (ours) 89.7…”

Section: Methods Stl10mentioning

confidence: 99%

“…IIC [49] 25.7 TSUC [50] 35.3 SCAN [16] 50.7 RUC [51] 54.3 LFER Ensemble [52] 56.1 ScatSimCLR [30] 63.8…”

Section: Methods Cifar20mentioning

confidence: 99%

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

Kinakh,

Drozdova,

Voloshynovskiy

2024

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…A well-crafted pretext task [26,27] is a cornerstone for achieving successful selfsupervised learning. The essence of such a task lies in setting up a specific objective within unlabeled data, enabling the model to glean meaningful representations from it.…”

Section: Self-supervised Learning With Cutpaste-mixmentioning

confidence: 99%

A Self-Supervised Model Based on CutPaste-Mix for Ductile Cast Iron Pipe Surface Defect Classification

Zhang,

Sun,

2023

Sensors

View full text Add to dashboard Cite

Online surface inspection systems have gradually found applications in industrial settings. However, the manual effort required to sift through a vast amount of data to identify defect images remains costly. This study delves into a self-supervised binary classification algorithm for addressing the task of defect image classification within ductile cast iron pipe (DCIP) images. Leveraging the CutPaste-Mix data augmentation strategy, we combine defect-free data with enhanced data to input into a deep convolutional neural network. Through Gaussian Density Estimation, we compute anomaly scores to achieve the classification of abnormal regions. Our approach has been implemented in real-world scenarios, involving equipment installation, data collection, and experimentation. The results demonstrate the robust performance of our method, in both the DCIP image dataset and practical field application, achieving an impressive 99.5 AUC (Area Under Curve). This presents a cost-effective means of providing data support for subsequent DCIP surface inspection model training.

show abstract

“…A base encoder and a projection head are then trained using a contrastive loss that we also leverage in one branch of ACM-Net, to maximize agreement between representations of these two views and minimize agreement with views originating from different images. Since training convergence of CL models may be difficult to achieve, and thus to regularize the training, ScatSimCLR [21] additionally regresses the augmentation parameters for each view. In ACM-Net, the CL training is regularized by adding a separate PL branch.…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Learning for Place Representation Generalization across Appearance Changes

Musallam¹,

Gaudillière²,

Aouada³

2023

Preprint

View full text Add to dashboard Cite

Visual place recognition is a key to unlocking spatial navigation for animals, humans and robots. While state-ofthe-art approaches are trained in a supervised manner and therefore hardly capture the information needed for generalizing to unusual conditions, we argue that self-supervised learning may help abstracting the place representation so that it can be foreseen, irrespective of the conditions. More precisely, in this paper, we investigate learning features that are robust to appearance modifications while sensitive to geometric transformations in a self-supervised manner. This dual-purpose training is made possible by combining the two self-supervision main paradigms, i.e. contrastive and predictive learning. Our results on standard benchmarks reveal that jointly learning such appearance-robust and geometry-sensitive image descriptors leads to competitive visual place recognition results across adverse seasonal and illumination conditions, without requiring any humanannotated labels 1 .

show abstract

ScatSimCLR: self-supervised contrastive learning with pretext task regularization for small-scale datasets

Cited by 8 publications

References 20 publications

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

A Self-Supervised Model Based on CutPaste-Mix for Ductile Cast Iron Pipe Surface Defect Classification

Self-Supervised Learning for Place Representation Generalization across Appearance Changes

Contact Info

Product

Resources

About