Handling Noisy Labels for Robustly Learning from Self-Training Data for Low-Resource Sequence Labeling

Abstract

In this paper, we address the problem of effectively self-training neural networks in a lowresource setting. Self-training is frequently used to automatically increase the amount of training data. However, in a low-resource scenario, it is less effective due to unreliable annotations created using self-labeling of unlabeled data. We propose to combine self-training with noise handling on the self-labeled data. Directly estimating noise on the combined clean training set and self-labeled data can lead to corruption of the clean data and hence, performs worse. Thus, we propose the Clean and Noisy Label Neural Network which trains on clean and noisy self-labeled data simultaneously by explicitly modelling clean and noisy labels separately. In our experiments on Chunking and NER, this approach performs more robustly than the baselines. Complementary to this explicit approach, noise can also be handled implicitly with the help of an auxiliary learning task. To such a complementary approach, our method is more beneficial than other baseline methods and together provides the best performance overall.

Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

“…To avoid this, it can be combined with label noise handling techniques. This pipeline has been shown to be effective for several NLP tasks (Lange et al, 2019;Paul et al, 2019;Wang et al, 2019;Chen et al, 2019;Mayhew et al, 2019), however, mostly for RNN based approaches. As we have seen in Section 4 that these have a lower baseline performance, we are interested in whether distant supervision is still useful for the better performing transformer models.…”

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

“…The noise in the labels can also be modeled. A common model is a confusion matrix estimating the relationship between clean and noisy labels (Fang and Cohn, 2016;Luo et al, 2017;Hedderich and Klakow, 2018;Paul et al, 2019;Lange et al, 2019a,c;Wang et al, 2019;Hedderich et al, 2021b). The classifier is no longer trained directly on the noisily-labeled data.…”

Extraction and Evaluation of Knowledge Entities from Scientific Documents

“…Journal of Data and Information Science the state-of-the-art one-deep learning-based (Paul et al, 2019;Yang et al, 2018), respectively.…”