Robust Loss Functions under Label Noise for Deep Neural Networks

Abstract

In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.

Deep Species Distribution Modeling From Sentinel-2 Image Time-Series: A Global Scale Analysis on the Orchid Family

“…In machine learning, such mismatch between labels and predictive data is called label noise (Frénay and Verleysen, 2013) and is actively studied (Ghosh et al, 2017;Lee et al, 2018). The strength of our dataset in counteracting this noise is its very large size, as demonstrated by Rolnick et al (2017).…”

Searching for Robustness: Loss Learning for Noisy Classification Tasks

“…Major existing robust losses include: mean absolute error (MAE), shown to be theoretically robust in [10], but hard to train in [54]; generalised cross-entropy (GCE) which attempts to be robust yet easy to train [54]; bi-temper [1], two-temperature [2], and Huber [15] motivated by heavy tailed outlier robustness; symmetric cross-entropy [45] motivated by reducing overfitting; and active-passive loss (APL) [31] which aims to balance over-and-underfitting for robust losses. These losses are all hand-designed based on various good motivations, but (as we will see in our evaluation) none provide reliably high performance empirically.…”

“…We compare our ARL with the standard cross-entropy (CE) baseline, as well as several strong alternative losses hand-designed for label-noise robustness: MAE: Mean Absolute Error was theoretically shown to be robust in [10]. GCE: [54] analysed MAE as hard to train, and proposed generalised cross-entropy to provide the best of CE and MAE; FW: [33] iteratively estimates the label noise transfer matrix, and trains the model corrected by the label noise estimate; SCE: [45] argued that symmetrising cross-entropy by adding reverse cross-entropy (RCE) improves label-noise robustness; Bootstrap: A classic method of replacing the noisy labels in training by the convex combination of the prediction and the given labels [37].…”

“…Existing robust alternatives MAE [10] and GCE [54] are explicitly motivated by softening this aggressive "correction" compared to CE. Although not explicitly motivated by this, SCE also softens the feedback as shown in the figure.…”

“…Existing hand-designed robust losses and our meta-learned robust loss. Top left: Conventional Cross-Entropy (CE); Top middle: Generalised Cross Entropy (GCE) [54]; Top right: Mean Absolute Error (MAE) [10]; Bottom left: label-smoothing [34]. Bottom middle: Symmetric Cross Entropy (SCE) [45].…”