Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

Zheltonozhskii, Evgenii; Baskin, Chaim; Mendelson, Avi; Bronstein, Alex; Litany, Or

doi:10.1109/wacv51458.2022.00046

Cited by 56 publications

(31 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The above results also imply when the noise rate is high such that Ỹ starts to become uninformative, dropping the noisy labels and focusing on obtaining the disentangled features helps with the identifiability of T (X). This observation helps explain recent successes in applying semi-supervised (Cheng et al, 2020a;Li et al, 2020;Nguyen et al, 2019) and self-supervised (Cheng et al, 2021;Zheltonozhskii et al, 2022;Ghosh & Lan, 2021) learning techniques to the problem of learning from noisy labels.…”

Section: Disentangled Featuresmentioning

confidence: 67%

Identifiability of Label Noise Transition Matrix

Liu¹

2022

Preprint

View full text Add to dashboard Cite

The noise transition matrix plays a central role in the problem of learning from noisy labels. Among many other reasons, a significant number of existing solutions rely on access to it. Estimating the transition matrix without using ground truth labels is a critical and challenging task. When label noise transition depends on each instance, the problem of identifying the instance-dependent noise transition matrix becomes substantially more challenging. Despite recent works proposing solutions for learning from instance-dependent noisy labels, we lack a unified understanding of when such a problem remains identifiable, and therefore learnable. This paper seeks to provide answers to a sequence of related questions: What are the primary factors that contribute to the identifiability of a noise transition matrix? Can we explain the observed empirical successes? When a problem is not identifiable, what can we do to make it so? We will relate our theoretical findings to the literature and hope to provide guidelines for developing effective solutions for battling instance-dependent label noise.

show abstract

Section: Disentangled Featuresmentioning

confidence: 67%

Identifiability of Label Noise Transition Matrix

Liu¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…By introducing soft labels, our CMW-Net-SL achieves superior performance to recent SOTA methods, DivideMix and ELR. Furthermore, by combining the self-supervised pretraining technique proposed in the C2D method [88], we further boost the performance. In Fig.…”

Section: Learning With Real-world Noisy Datasetsmentioning

confidence: 99%

“…Following the standard protocol [37], we test the trained model on the WebVision validation set and the ImageNet validation set. Following C2D [88], we used ResNet-50 architecture as the classifier network for training. For self-supervised pre-training, we directly use the pretrained self-supervised models released at https://github.com/ContrastToDivide/C2D, which is based on the SimCLR implementation.…”

Section: Mini-webvisionmentioning

confidence: 99%

CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning

Shen¹,

Yuan²,

Meng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern deep neural networks (DNNs) can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. Most current methods, however, require to manually pre-specify the weighting schemes as well as their additional hyper-parameters relying on the characteristics of the investigated problem and training data. This makes them fairly hard to be generally applied in practical scenarios, due to their significant complexities and inter-class variations of data bias situations. To address this issue, we propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data. Specifically, by seeing each training class as a separate learning task, our method aims to extract an explicit weighting function with sample loss and task/class feature as input, and sample weight as output, expecting to impose adaptively varying weighting schemes to different sample classes based on their own intrinsic bias characteristics. Synthetic and real data experiments substantiate the capability of our method on achieving proper weighting schemes in various data bias cases, like the class imbalance, feature-independent and dependent label noise scenarios, and more complicated bias scenarios beyond conventional cases. Besides, the task-transferability of the learned weighting scheme is also substantiated, by readily deploying the weighting function learned on relatively smaller-scale CIFAR-10 dataset on much larger-scale full WebVision dataset. A performance gain can be readily achieved compared with previous state-of-the-art ones without additional hyper-parameter tuning and meta gradient descent step. The general availability of our method for multiple robust deep learning issues, including partial-label learning, semi-supervised learning and selective classification, has also been validated.

show abstract

“…Overview. Following the two-stage learning style [13,32,41,65], our aim is to induce robust pre-trained representations of instances for learning with noisy labels.…”

Section: Selective-supervised Contrastive Learningmentioning

confidence: 99%

“…Recent works [7,8,13,18,65] show that, working in a pair-wise manner, contrastive learning (CL) methods can bring good latent representations to help following tasks. Based on whether supervised information is provided, the CL methods can be grouped into supervised contrastive learning (Sup-CL) [27] and unsupervised contrastive learning (Uns-CL) [7,8,18].…”

Section: Introductionmentioning

confidence: 99%

Selective-Supervised Contrastive Learning with Noisy Labels

Li¹,

Xia²,

Ge³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep networks have strong capacities of embedding data into latent representations and finishing following tasks. However, the capacities largely come from high-quality annotated labels, which are expensive to collect. Noisy labels are more affordable, but result in corrupted representations, leading to poor generalization performance. To learn robust representations and handle noisy labels, we propose selective-supervised contrastive learning (Sel-CL) in this paper. Specifically, Sel-CL extend supervised contrastive learning (Sup-CL), which is powerful in representation learning, but is degraded when there are noisy labels. Sel-CL tackles the direct cause of the problem of Sup-CL. That is, as Sup-CL works in a pair-wise manner, noisy pairs built by noisy labels mislead representation learning. To alleviate the issue, we select confident pairs out of noisy ones for Sup-CL without knowing noise rates. In the selection process, by measuring the agreement between learned representations and given labels, we first identify confident examples that are exploited to build confident pairs. Then, the representation similarity distribution in the built confident pairs is exploited to identify more confident pairs out of noisy pairs. All obtained confident pairs are finally used for Sup-CL to enhance representations. Experiments on multiple noisy datasets demonstrate the robustness of the learned representations by our method, following the state-of-the-art performance. Source codes are available at https://github.com/ShikunLi/Sel-CL

show abstract

Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

Cited by 56 publications

References 20 publications

Identifiability of Label Noise Transition Matrix

Identifiability of Label Noise Transition Matrix

CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning

Selective-Supervised Contrastive Learning with Noisy Labels

Contact Info

Product

Resources

About