Correcting a PoS-tagged corpus using three complementary methods

Abstract

The quality of the part-of-speech (PoS) annotation in a corpus is crucial for the development of PoS taggers. In this paper, we experiment with three complementary methods for automatically detecting errors in the PoS annotation for the Icelandic Frequency Dictionary corpus. The first two methods are language independent and we argue that the third method can be adapted to other morphologically complex languages. Once possible errors have been detected, we examine each error candidate and hand-correct the corresponding PoS tag if necessary. Overall, based on the three methods, we handcorrect the PoS tagging of 1,334 tokens (0.23% of the tokens) in the corpus. Furthermore, we re-evaluate existing state-ofthe-art PoS taggers on Icelandic text using the corrected corpus.

Semi-Automated Resolution of Inconsistency for a Harmonized Multiword Expression and Dependency Parse Annotation

“…Boyd et al (2007) employed an alignment-based approach to assess differences in the annotation of n-gram word sequences in order to establish the likelihood of error occurrence. Other work in the syntactic inconsistency detection domain includes those related to POS tagging (Loftsson, 2009;Eskin, 2000;Ma et al, 2001) and parse structure (Ule and Simov, 2004 (2) ambiguity associated with the assessment of hard cases. While annotation errors apply to situations where a correct label can be applied but is not done so, hard cases are those where the correct label is inherently difficult to assign, and can be particularly relevant to certain classes of MWEs.…”

Detecting annotation noise in automatically labelled data

“…Quite a bit of work has been devoted to the identifcation of errors in manually annotated corpora (Eskin, 2000;van Halteren, 2000;Kveton and Oliva, 2002;Dickinson and Meurers, 2003;Loftsson, 2009;Ambati et al, 2011).…”

“…The query-by-committee strategy calls to mind previous work on error detection in manually labelled text that made use of disagreements between the predictions of a classifier ensemble and the manually assigned tag, to identify potential annotation errors in the data (Loftsson, 2009). This approach works surprisingly well, and the tradeoff between precision and recall can be balanced by adding a threshold (i.e.…”

What I’ve learned about annotating informal text (and why you shouldn’t take my word for it)

“…I hope future research will succeed at making this process more empirical and more predictable (see also Hovy and Lavid, 2010;Garrette and Baldridge, 2013). There is a great deal more to discover with regard to understanding the range of text varieties (Baldwin et al, 2013), building statistical models of annotator bias (Snow et al, 2008;Hovy et al, 2013;Passonneau and Carpenter, 2014), automatically detecting inconsistencies in linguistic data (Dickinson and Meurers, 2003;Loftsson, 2009;Kato and Matsubara, 2010), and bringing extrinsic models into the annotation loop (Baldridge and Osborne, 2004;Baldridge and Palmer, 2009;Settles, 2012).…”