Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?

Abstract

Abstract. I examine what would be necessary to move part-of-speech tagging performance from its current level of about 97.3% token accuracy (56% sentence accuracy) to close to 100% accuracy. I suggest that it must still be possible to greatly increase tagging performance and examine some useful improvements that have recently been made to the Stanford Part-of-Speech Tagger. However, an error analysis of some of the remaining errors suggests that there is limited further mileage to be had either from better machine learning or better features in a discriminative sequence classifier. The prospects for further gains from semisupervised learning also seem quite limited. Rather, I suggest and begin to demonstrate that the largest opportunity for further progress comes from improving the taxonomic basis of the linguistic resources from which taggers are trained. That is, from improved descriptive linguistics. However, I conclude by suggesting that there are also limits to this process. The status of some words may not be able to be adequately captured by assigning them to one of a small number of categories. While conventions can be used in such cases to improve tagging consistency, they lack a strong linguistic basis. Isn't Part-of-Speech Tagging a Solved Task?At first glance, current part-of-speech taggers work rapidly and reliably, with per-token accuracies of slightly over 97% [1][2][3][4]. Looked at more carefully, the story is not quite so rosy. This evaluation measure is easy both because it is measured per-token and because you get points for every punctuation mark and other tokens that are not ambiguous. It is perhaps more realistic to look at the rate of getting whole sentences right, since a single bad mistake in a sentence can greatly throw off the usefulness of a tagger to downstream tasks such as dependency parsing. Current good taggers have sentence accuracies around 55-57%, which is a much more modest score. Accuracies also drop markedly when there are differences in topic, epoch, or writing style between the training and operational data.Still, the perception has been that same-epoch-and-domain part-of-speech tagging is a solved problem, and its accuracy cannot really be pushed higher. I

Extracting conceptual models from user stories with Visual Narrator

“…For example, the spaCy tagger would find the compound events_section in the sentence ''I keep the events section'', but it would miss it in the sentence ''I can keep the events section''. This is due to the fact that modern NLP taggers are statistical and try to find the most probable tagging in many situations [40]. Furthermore, we do not yet support compounds that consist of three or more nouns, such as ''profile page statistics''.…”

Data properties and the performance of sentiment classification for electronic commerce applications

“…In corpus linguistic and computational linguistics, part-of-speech tagging (POS tagging) is a process that makes up a word in a text (corpus) as corresponding to a particular part of speech, such as noun, verb, adjective, etc. The automatic POS tagging has a long history in computational linguistic studies and now its tagging accuracy reached to over 97% 2 (Manning 2011;Toutanova et al 2003). POS tagged document vectors can be easily scored by comparing adjectives/adverbs/verbs in documents and those in SentiWordNet that contains the polarity scores of words (for example, adjective 'bad' has −0.625 and 'worst' has −0.75 in SentiWordNet).…”

Adapting a Constituency Parser to User-Generated Content in Polish Opinion Mining

“…Although the machine-learning techniques seem to prevail in the aforementioned studies, we have been inspired by the rule-based methods of parser and POS-tagger tuning, much as fixing of Penn Treebank errors with deterministic rules as described by Manning in [20], especially for the purposes of unknown vocabulary recognition. In his study, Manning analyzes the 100 most recurrent parser errors and offers rule-based solutions to specific problems that hinder a parser's good performance, such as lexicon gaps, unknown vocabulary, difficult linguistics, or having no standard to learn from.…”