Small-Vote Sample Selection for Label-Noise Learning

Abstract

The small-loss criterion is widely used in recent label-noise learning methods. However, such a criterion only considers the loss of each training sample in a mini-batch but ignores the loss distribution in the whole training set. Moreover, the selection of clean samples depends on a heuristic clean data rate. As a result, some noisy-labeled samples are easily identified as clean ones, and vice versa. In this paper, we propose a novel yet simple sample selection method, which mainly consists of a Hierarchical Voting Scheme (HVS) and an Adaptive Clean data rate Estimation Strategy (ACES), to accurately identify clean samples and noisy-labeled samples for robust learning. Specifically, we propose HVS to effectively combine the global vote and the local vote, so that both epoch-level and batch-level information is exploited to assign a hierarchical vote for each mini-batch sample. Based on HVS, we further develop ACES to adaptively estimate the clean data rate by leveraging a 1D Gaussian Mixture Model (GMM). Experimental results show that our proposed method consistently outperforms several state-of-theart label-noise learning methods on both synthetic and real-world noisy benchmark datasets.

A Survey of Historical Learning: Learning Models with Learning History

“…Inside, cyclical training records the historical loss of each sample, then calculates and ranks every sample's normalized average loss to recognize the noisy data with a higher score. Later, some works introduce the value of historical loss to determine whether the data is with noisy label [142]- [145]. To achieve a better performance of cleaning samples, Xu et al [142] also use the historical loss distribution to handle the issue of varying scales of loss distributions across different training periods.…”

“…Later, some works introduce the value of historical loss to determine whether the data is with noisy label [142]- [145]. To achieve a better performance of cleaning samples, Xu et al [142] also use the historical loss distribution to handle the issue of varying scales of loss distributions across different training periods. Although average loss has shown promising results [141], the existence of outliers in the training data can result in deviations when cleaning the data.…”

Temporal Ensembling-based Deep k-Nearest Neighbours for Learning with Noisy Labels

“…Several methods detect clean samples as the ones for which a small loss is obtained [3, 2], motivated by the observation that neural networks first learn the correct labels and only afterwards overfit the noisy ones. In [2], scores are assigned to small-loss samples using so-called local votes, that take into account only the current batch and global votes, which are computed using the training history. Co-teaching [3] is based on the smallloss selection strategy, but uses two networks which select clean instances for each other.…”

“…Then, a constant number of samples is selected. We note, however, that other strategies are possible, such as fitting a Gaussian mixture model on the obtained scores, as in [2] or defining adaptive perclass thresholds as in [5]. We study two variations of the proposed approach, by considering two types of initialization: zero-initialization and autoencoder (AE) initialization.…”

“…While obtaining such high quality labels can be costly, annotations retrieved through automatic but imprecise methods are usually much cheaper. Due to this reason, robust learning in the presence of label noise has a great practical importance [1,2].…”