An Effective Label Noise Model for

Abstract

Because large, human-annotated datasets suffer from labeling errors, it is crucial to be able to train deep neural networks in the presence of label noise. While training image classification models with label noise have received much attention, training text classification models have not. In this paper, we propose an approach to training deep networks that is robust to label noise. This approach introduces a non-linear processing layer (noise model) that models the statistics of the label noise into a convolutional neural network (CNN) architecture. The noise model and the CNN weights are learned jointly from noisy training data, which prevents the model from overfitting to erroneous labels. Through extensive experiments on several text classification datasets, we show that this approach enables the CNN to learn better sentence representations and is robust even to extreme label noise. We find that proper initialization and regularization of this noise model is critical. Further, by contrast to results focusing on large batch sizes for mitigating label noise for image classification, we find that altering the batch size does not have much effect on classification performance.

Why Do Document-Level Polarity Classifiers Fail?

“…Class noise is considered to be more harmful than attribute noise (Frenay and Verleysen, 2014), but it is easier to detect (Van Hulse et al, 2007). Thus, class noise is more often addressed in the literature (Gupta and Gupta, 2019), where several studies analyzed its impact in classification tasks and how to address it (Natarajan et al, 2013;Hendrycks et al, 2018;Liu et al, 2017;Rehbein and Ruppenhofer, 2017;Jindal et al, 2019). In NLP, attribute noise are unintended errors in text, which can come from failures in automatic character recognition processes (Vinciarelli, 2005) or naturally while writing the text in the form of errors in language rules, such as typos, grammatical errors, improper punctuation, irrational capitalization and abbreviations (Agarwal et al, 2007;Contractor et al, 2010;Dey and Haque, 2009;Florian et al, 2010;Michel and Neubig, 2018).…”

Training Dynamic based data filtering may not work for NLP datasets

“…Several efforts have been made to account for noise in the data and prevent the model from memorizing wrong examples without actually identifying and removing such examples from the training set. Li et al (2020) replaces the last layer of models trained on noisy data with a linear layer trained on a small set of clean data, Jindal et al (2019) adds a non-linear noise-modeling layer on top of the target text-classification model. Kang and Hashimoto (2020) Other types of Noise in text data.…”

Improvement of Ancient Shui Character Recognition Model Based on Convolutional Neural Network

“…Image classication has been dominated by variants of convolutional neural networks [12], since it has high learning capacity and steady performance [13], although the recognition accuracy of the convolutional neural network model in image recognition is very high, it is not competitive enough in some fields like recommendation [14]. It requires many labelled image samples for providing training support, current research on image classification mainly relies on manual labelling [15], [16]. For example, most of the ImageNet databases that are used in image research are manually labelled [11].…”