Classifier‐Guided Visual Correction of Noisy Labels for Image Classification Tasks

Abstract

Training data plays an essential role in modern applications of machine learning. However, gathering labeled training data is time‐consuming. Therefore, labeling is often outsourced to less experienced users, or completely automated. This can introduce errors, which compromise valuable training data, and lead to suboptimal training results. We thus propose a novel approach that uses the power of pretrained classifiers to visually guide users to noisy labels, and let them interactively check error candidates, to iteratively improve the training data set. To systematically investigate training data, we propose a categorization of labeling errors into three different types, based on an analysis of potential pitfalls in label acquisition processes. For each of these types, we present approaches to detect, reason about, and resolve error candidates, as we propose measures and visual guidance techniques to support machine learning users. Our approach has been used to spot errors in well‐known machine learning benchmark data sets, and we tested its usability during a user evaluation. While initially developed for images, the techniques presented in this paper are independent of the classification algorithm, and can also be extended to many other types of training data.

Towards Better Caption Supervision for Object Detection

“…To handle label noise in such datasets, Xiang et al [17] tightly integrated a scalable trusted-item-based correction algorithm with an incremental t-SNE algorithm to support an iterative refinement procedure. Bäuerle et al [18] proposed three error detection measures, class interpretation error score, instance interpretation error score, and similarity error score, and leveraged them to correct label errors.…”

Visual Identification of Problematic Bias in Large Label Spaces

“…While there have been many visualization approaches targeted towards large datansets, such as active learning [28,30,35], interactive labeling [6,7], labeling process improvements [15,40], and data cleaning [11,59], they are all used during the labeling process. Fairness analysis, on the contrary, happens either after dataset collection has been completed (data fairness), or even after model training has been finished (model fairness).…”

“…Here users should be able to confirm or reject potentially problematic bias, thus cleaning up the data for actionable interventions. Drawing inspiration from other correction approaches in the machine learning domain [11,59], we propose to enable marking of problematic labels, and removal of unproblematic ones from the visualizations. Using these filtering approaches, which are applied from coarse (sensitive attribute) to fine (correlation values), and providing useful overview visualizations throughout, we follow Shneiderman's Mantra [51] of providing an overview first, then being able to zoom and filter.…”

A survey of visual analytics techniques for machine learning

“…After correction, the classifier is refined using the corrected labels, and a new round of correction starts. Bäuerle et al [16] developed three classifier-guided measures to detect data errors. Data errors are then presented in a matrix and a scatter plot, allowing experts to reason about and resolve errors.…”