Better Self-training for Image Classification Through Self-supervision

Sahito, Attaullah; Frank, Eibe; Pfahringer, Bernhard

doi:10.1007/978-3-030-97546-3_52

Cited by 5 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Self-training has shown promising progress in many domains including vision [2,3,20], NLP [21], and speech [22]. Our method is more closely related to the self-training approaches proposed for semi-supervised learning [23,24,25], where pseudo-labels on unlabeled data are used as training targets.…”

Section: Related Workmentioning

confidence: 99%

Masked Unsupervised Self-training for Zero-shot Image Classification

Li¹,

Savarese²,

Hoi³

2022

Preprint

View full text Add to dashboard Cite

State-of-the-art computer vision models are mostly trained with supervised learning using human-labeled images, which limits their scalability due to the expensive annotation cost. While self-supervised representation learning has achieved impressive progress, it still requires a second stage of finetuning on labeled data. On the other hand, models pre-trained with large-scale text-image supervision (e.g., CLIP) have enabled zero-shot transfer to downstream image classification tasks. However, the zero-shot performance of CLIP-like models are often insufficient for real-world adoption. In this paper, we aim to leverage the abundant unlabeled data to improve the performance of a pre-trained zero-shot classifier on downstream tasks. We propose Masked Unsupervised Self-Training (MUST), a new approach which leverages two different and complimentary sources of supervision: pseudo-labels and raw images. MUST jointly optimizes three objectives to learn both class-level global feature and pixel-level local feature and enforces a regularization between the two. We demonstrate the efficacy of MUST on 8 downstream tasks across a variety of domains, where it improves upon CLIP by a large margin and narrows the performance gap between unsupervised and supervised classification. For instance, MUST achieves a zero-shot top-1 accuracy of 77.7% on ImageNet using ViT-B, +9.4% higher than CLIP. Our code is available at https://github.com/salesforce/MUST.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Masked Unsupervised Self-training for Zero-shot Image Classification

Li¹,

Savarese²,

Hoi³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Results we have shown on training with random labels are reminiscent of works in the field of semi-supervised training. Here, the goal is to find auto-generated "pretext tasks" such that training on them lead to the model learning good representations which are useful for downstream tasks [22,16,5]. This is often done with auto-generated labels such as predicting image rotations [7].…”

Section: Related Workmentioning

confidence: 99%

What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective

Chowers¹,

Weiss²

2022

Preprint

View full text Add to dashboard Cite

It has previously been observed that the filters learned in the first layer of a CNN are qualitatively similar for different networks and tasks. We extend this finding and show a high quantitative similarity between filters learned by different networks. We consider the CNN filters as a filter bank and measure the sensitivity of the filter bank to different frequencies. We show that the sensitivity profile of different networks is almost identical, yet far from initialization. Remarkably, we show that it remains the same even when the network is trained with random labels. To understand this effect, we derive an analytic formula for the sensitivity of the filters in the first layer of a linear CNN. We prove that when the average patch in images of the two classes is identical, the sensitivity profile of the filters in the first layer will be identical in expectation when using the true labels or random labels and will only depend on the second-order statistics of image patches. We empirically demonstrate that the average patch assumption holds for realistic datasets. Finally we show that the energy profile of filters in nonlinear CNNs is highly correlated with the energy profile of linear CNNs and that our analysis of linear networks allows us to predict when representations learned by state-of-the-art networks trained on benchmark classification tasks will depend on the labels.Preprint. Under review.

show abstract

“…This was later extended to include a measure of confidence in (Shi et al 2018). Closely related is the concept of selftraining which iteratively integrates into training the most confident of these pseudo-labeled samples and repeats (Dong and Schäfer 2011;Sahito, Frank, and Pfahringer 2021;Xie et al 2020b). These techniques can become unstable when pseudo-label error accumulates across iterations.…”

Section: Introductionmentioning

confidence: 99%

Contrastive Credibility Propagation for Reliable Semi-supervised Learning

Kutt,

Ramteke,

Mignot

et al. 2024

AAAI

View full text Add to dashboard Cite

Producing labels for unlabeled data is error-prone, making semi-supervised learning (SSL) troublesome. Often, little is known about when and why an algorithm fails to outperform a supervised baseline. Using benchmark datasets, we craft five common real-world SSL data scenarios: few-label, open-set, noisy-label, and class distribution imbalance/misalignment in the labeled and unlabeled sets. We propose a novel algorithm called Contrastive Credibility Propagation (CCP) for deep SSL via iterative transductive pseudo-label refinement. CCP unifies semi-supervised learning and noisy label learning for the goal of reliably outperforming a supervised baseline in any data scenario. Compared to prior methods which focus on a subset of scenarios, CCP uniquely outperforms the supervised baseline in all scenarios, supporting practitioners when the qualities of labeled or unlabeled data are unknown.

show abstract

Better Self-training for Image Classification Through Self-supervision

Cited by 5 publications

References 11 publications

Masked Unsupervised Self-training for Zero-shot Image Classification

Masked Unsupervised Self-training for Zero-shot Image Classification

What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective

Contrastive Credibility Propagation for Reliable Semi-supervised Learning

Contact Info

Product

Resources

About