Named Entity Recognition with Partially Annotated Training Data

Abstract

Supervised machine learning assumes the availability of fully-labeled data, but in many cases, such as low-resource languages, the only data available is partially annotated. We study the problem of Named Entity Recognition (NER) with partially annotated training data in which a fraction of the named entities are labeled, and all other tokens, entities or otherwise, are labeled as non-entity by default. In order to train on this noisy dataset, we need to distinguish between the true and false negatives. To this end, we introduce a constraintdriven iterative algorithm that learns to detect false negatives in the noisy set and downweigh them, resulting in a weighted training set. With this set, we train a weighted NER model. We evaluate our algorithm with weighted variants of neural and non-neural NER models on data in 8 languages from several language and script families, showing strong ability to learn from partial data. Finally, to show real-world efficacy, we evaluate on a Bengali NER corpus annotated by non-speakers, outperforming the prior state-of-the-art by over 5 points F1.

An Imitation Game for Learning Semantic Parsers from User Interaction

“…Finally, we believe our algorithm can be applied to save annotation effort for other NLP tasks, especially the low-resource ones (Mayhew et al, 2019).…”

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

“…Partial annotation learning Nooralahzadeh et al, 2019;Cao et al, 2019) takes this into account explicitly. Related approaches learn latent variables , use constrained binary learning (Mayhew et al, 2019) or construct a loss assuming that only unlabeled positive instances exist .…”

“…For instance, Garrette and Baldridge (2013) obtain labeled data from non-native-speakers and without a quality control on the manual annotations. This can be taken even further by employing annotators who do not speak the low-resource language (Mayhew and Roth, 2018;Mayhew et al, 2019;Tsygankova et al, 2020). Nekoto et al (2020) take the opposite direction, integrating speakers of low-resource languages without formal training into the model development process in an approach of participatory research.…”

Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances

“…There has been a small amount of work using non-speaker annotations (Mayhew et al, 2019a), but mainly as an application of a technique, falling short of the exhaustive study in this paper.…”

“…We recognize that these annotations are missing many entities. Following recent work on partial annotations, we use an iterative method from (Mayhew et al, 2019a) called Constrained Binary Learning (CBL) that detects unmarked tokens likely to be entities and down-weights them in training. Subsequent results reported use this method on all FS and NS annotations.…”