Constructing an anaphorically annotated corpus with non-experts

Chamberlain, Jon; Kruschwitz, Udo; Poesio, Massimo

doi:10.3115/1699765.1699774

Cited by 16 publications

(13 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There is a clear difference in quality when we look at the difficulty of the task in GWAP [12]. Looking separately at the agreement on each type of markable annotation in PD (see Table 1), we see that the figures for a discourse-new (DN) annotation are very close for all three comparisons, and well over 90%.…”

Section: Task Difficultymentioning

confidence: 87%

“…Both games continue to collect data but results reported here are from December 2008 to February 2012 or are from previously published papers [12,13,15,59]. …”

Section: Phrase Detectivesmentioning

confidence: 99%

“…Text used in PD comes from two main domains: Wikipedia articles selected from the 'Featured Articles' page 12 and the page of 'Unusual Articles' 13 ; and narrative text from Project Gutenberg 14 including simple short stories (e.g., Aesop's Fables, Grimm's Fairy Tales, Beatrix Potter's tales) and more advanced narratives such as several Sherlock Holmes stories by A. Conan-Doyle, Alice in Wonderland, and several short stories by Charles Dickens.…”

Section: Phrase Detectivesmentioning

confidence: 99%

See 2 more Smart Citations

Using Games to Create Language Resources: Successes and Limitations of the Approach

Chamberlain

Fort

Kruschwitz

et al. 2013

The People’s Web Meets NLP

Self Cite

View full text Add to dashboard Cite

One of the more novel approaches to collaboratively creating language resources in recent years is to use online games to collect and validate data. The most significant challenges collaborative systems face are how to train users with the necessary expertise and how to encourage participation on a scale required to produce high quality data comparable with data produced by "traditional" experts. In this chapter we provide a brief overview of collaborative creation and the different approaches that have been used to create language resources, before analysing games used for this purpose. We discuss some key issues in using a gaming approach, including task design, player motivation and data quality, and compare the costs of each approach in terms of development, distribution and ongoing administration. In conclusion, we summarise the benefits and limitations of using a gaming approach to resource creation and suggest key considerations for evaluating its utility in different research scenarios.

show abstract

Section: Task Difficultymentioning

confidence: 87%

“…Both games continue to collect data but results reported here are from December 2008 to February 2012 or are from previously published papers [12,13,15,59]. …”

Section: Phrase Detectivesmentioning

confidence: 99%

Section: Phrase Detectivesmentioning

confidence: 99%

See 1 more Smart Citation

Using Games to Create Language Resources: Successes and Limitations of the Approach

Chamberlain

Fort

Kruschwitz

et al. 2013

The People’s Web Meets NLP

Self Cite

View full text Add to dashboard Cite

show abstract

“…39 Ainsi, l'accord inter-annotateur entre les annotateurs experts et les joueurs sur Phrase Detectives se situe entre 0,7 et 0,8 d'accord observé [9]. Ces chiffres sont très proches de ceux que nous obtenons sur le jeu ZombiLingo : des résultats préliminaires, récoltés sur une version encore non finalisée du jeu, montrent une moyenne d'exactitude de 0,86 sur les dix meilleurs joueurs (en enlevant le meilleur et le moins bon) et d'environ 0,9 sur la totalité des joueurs 17 (dont la plupart ne jouent que des relations simples).…”

Section: Corela Hs-21 | 2017unclassified

Experts ou (foule de) non-experts ? la question de l’expertise des annotateurs vue de la myriadisation (crowdsourcing)

Fort¹

2017

corela

View full text Add to dashboard Cite

“…Online collaborative annotation (Chamberlain et al, 2009) has recently emerged as an increasingly viable alternative to more conventional approaches for developing tagged corpora. In a linguistic setting, online collaboration has been used for a wide variety of tasks ranging from syntactic annotation of Latin and Ancient Greek texts (Bamman et.…”

Section: Introductionmentioning

confidence: 99%

Supervised collaboration for syntactic annotation of Quranic Arabic

Dukes

Atwell

Habash

2011

Lang Resources & Evaluation

View full text Add to dashboard Cite

The Quranic Arabic Corpus (http://corpus.quran.com) is a collaboratively constructed linguistic resource initiated at the University of Leeds, with multiple layers of annotation including part-of-speech tagging, morphological segmentation and syntactic analysis using dependency grammar (Dukes & Buckwalter, 2010). The motivation behind this work is to produce a resource that enables further analysis of the Quran, the 1,400 year-old central religious text of Islam. This project contrasts with other Arabic treebanks by providing a deep linguistic model based on the historical traditional grammar known as i'r b ( ). By adapting this well-known canon of Quranic grammar into a familiar tagset, it is possible to encourage online annotation by Arabic linguists and Quranic experts.This article presents a new approach to linguistic annotation of an Arabic corpus: online supervised collaboration using a multi-stage approach. The different stages include automatic rule-based tagging, initial manual verification, and online supervised collaborative proofreading. A popular website attracting thousands of visitors per day, the Quranic Arabic Corpus has approximately 100 unpaid volunteer annotators each suggesting corrections to existing linguistic tagging. To ensure a high-quality resource, a small number of expert annotators are promoted to a supervisory role, allowing them to review or veto suggestions made by other collaborators. The Quran also benefits from a large body of existing historical grammatical analysis, which may be leveraged during this review. In this paper we evaluate and report on the effectiveness of the chosen annotation methodology. We also discuss the unique challenges of annotating Quranic Arabic online and describe the custom linguistic software used to aid collaborative annotation.

show abstract

Constructing an anaphorically annotated corpus with non-experts

Cited by 16 publications

References 9 publications

Using Games to Create Language Resources: Successes and Limitations of the Approach

Using Games to Create Language Resources: Successes and Limitations of the Approach

Experts ou (foule de) non-experts ? la question de l’expertise des annotateurs vue de la myriadisation (crowdsourcing)

Supervised collaboration for syntactic annotation of Quranic Arabic

Contact Info

Product

Resources

About