Identifying Mislabeled Instances in Classification Datasets

Abstract

A key requirement for supervised machine learning is labeled training data, which is created by annotating unlabeled data with the appropriate class. Because this process can in many cases not be done by machines, labeling needs to be performed by human domain experts. This process tends to be expensive both in time and money, and is prone to errors. Additionally, reviewing an entire labeled dataset manually is often prohibitively costly, so many real world datasets contain mislabeled instances.To address this issue, we present in this paper a nonparametric end-to-end pipeline to find mislabeled instances in numerical, image and natural language datasets. We evaluate our system quantitatively by adding a small number of label noise to 29 datasets, and show that we find mislabeled instances with an average precision of more than 0.84 when reviewing our system's top 1% recommendation. We then apply our system to publicly available datasets and find mislabeled instances in CIFAR-100, Fashion-MNIST, and others. Finally, we publish the code and an applicable implementation of our approach.

Prevalence of neural collapse during the terminal phase of deep learning training

“…On the horizontal axis of each cell is the epoch of training. For each dataset–network combination, the red vertical line marks the beginning of the effective beginning of TPT (i.e., the epoch when the training accuracy reaches 99.6% for ImageNet and 99.9% for the remaining datasets); we do not use 100% as it has been reported ( 24 – 26 ) that several of these datasets contain inconsistencies and mislabels, which sometimes prevent absolute memorization. Additionally, orange lines denote measurements on the network classifier, while blue lines denote measurements on the activation class means.…”

Curating Cyberbullying Datasets: a Human-AI Collaborative Approach

“…A more recent study by Müller and Markert (2019) introduced a pipeline that can identify mislabeled data in numerical, image, and natural language datasets. The efficacy of their pipeline was evaluated by introducing noisy data, or data that was intentionally changed to be different from its original label, in an amount of 1%, 2%, or 3%, into 29 well-known realworld and synthetic classification datasets.…”

“…Using machine learning-based majority voting or consensus filtering methods has been applied extensively in prior research for classification datasets focused on topics such as finance, medical diagnosis, and news media (Brodley & Friedl, 1999;Ekambaram et al, 2017;Guan et al, 2011;Müller & Markert, 2019;Samami et al, 2020). However, to the best of our knowledge, these methods have not yet been applied to cyberbullying datasets.…”

Training Dynamic based data filtering may not work for NLP datasets

“…Bhardwaj et al (2010) uses statistical methods to find annotators whose annotations differ considerably from the remaining annotators and use manual inspection to decide the verdict for samples annotated by these annotators. Müller and Markert (2019) classifies training samples with the lowest gold label probabilities on a robust classifier as potentially mislabelled followed by their manual review for final decision. Zhang and Sugiyama (2021) detects samples with erroneous labels using an instance-dependent noise model along with instance-based embedding to capture instance-specific label corruption.…”