Training-ValueNet: Data Driven Label Noise Cleaning on Weakly-Supervised Web Images

Abstract

Manually labelling new datasets for image classification remains expensive and time-consuming. A promising alternative is to utilize the abundance of images on the web for which search queries or surrounding text offers a natural source of weak supervision. Unfortunately the label noise in these datasets has limited their use in practice. Several methods have been proposed for performing unsupervised label noise cleaning, the majority of which use outlier detection to identify and remove mislabeled images. In this paper, we argue that outlier detection is an inherently unsuitable approach for this task due to major flaws in the assumptions it makes about the distribution of mislabeled images. We propose an alternative approach which makes no such assumptions. Rather than looking for outliers, we observe that mislabeled images can be identified by the detrimental impact they have on the performance of an image classifier. We introduce training-value as an objective measure of the contribution each training example makes to the validation loss. We then present the training-value approximation network (Training-ValueNet) which learns a mapping between each image and its training-value. We demonstrate that by simply discarding images with a negative training-value, Training-ValueNet is able to significantly improve classification performance on a held-out test set, outperforming the state of the art in outlier detection by a large margin.

Human-Expert-Level Brain Tumor Detection Using Deep Learning with Data Distillation and Augmentation

“…Noisy labels are posing a non-trivial problem in deep model learning when an increasing ability to fit noise is accompanied with deeper layers. Given the ubiquity and importance of coping with noisy labeling, many works have been devoted to combat this problem [4,5,17,18,19]. One promising direction is to learn from a small set of clean labeled data and then use them to update the network [17,4,19].…”

“…One promising direction is to learn from a small set of clean labeled data and then use them to update the network [17,4,19]. Another direction is to design models that could learn directly with noisy labels [17,18,20]. In [4], an auxiliary model is trained with a small but clean data set, which was manually labeled by human experts.…”

Human-Expert-Level Brain Tumor Detection Using Deep Learning with Data Distillation And Augmentation

“…Additionally, labels are only provided per patient and not per voxel, which could introduce labeling noise as spectra from the tumor-affected hemisphere can be falsely labeled as "tumor" even though they contain healthy brain tissue. Given the ubiquity and importance of coping with noisy labeling, many works on this topic have been published [8][9][10][11]. Starting learning from a small set of expert validated labels is one promising direction [8,10].…”

“…Starting learning from a small set of expert validated labels is one promising direction [8,10]. Another direction is to design models that learn directly with noisy labels [10,11]. For example, [10] uses a co-teaching framework where two DNNs were trained simultaneously with noisy labeling, or [11] discards samples that contribute negatively to the training performance.…”

“…1. Notably, it does not require human supervision to carefully annotate a small data set as prototype samples, but rather learns directly on the noisy labeled data as in [10,11]. Note that in this work, the distillation network and the final classifier have the same structure, but this need not be the case.…”