Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies

Abstract

ii IntroductionThis volume contains papers describing systems submitted to the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies and an overview paper summarizing the task, its features, evaluation methodology for the main and additional metrics, and some interesting observations about the submitted systems and the task as a whole.This Shared Task (http://universaldependencies.org/conll17/) can be seen as an extension of the CoNLL 2007 Shared Task on parsing, but there are many important differences that make this year's task unique with several "firsts". Most importantly, the data for this task come from the Universal Dependencies project (http://universaldependencies.org), which provides annotated treebanks for a large number of languages using the same annotation scheme for all of them. In the shared task setting, this allows for more meaningful comparison between systems as well as languages, since differences are much more likely due to true parser differences rather than differences caused by annotation schemes. In addition, the number of languages for which training data were available is unprecedented for a single shared task: a total of 64 treebanks in 45 languages have been provided for training the systems. Additional data have been provided too, as were some baseline systems for those who wanted to try only some particular aspect of parsing. Overall, the task can be described as "closed", since only pre-approved data could be used.For evaluation, there were 81 datasets (standard datasets for the treebank languages provided for training, plus more test sets in known languages, but based on a specially created and annotated parallel corpus, and four surprise language test sets). Participants had to process all the test sets. The TIRA platform has been used for evaluation, as was the case already for the CoNLL 2015 and 2016 Shared Tasks, meaning that participants had to provide their code on a designated virtual machine to be run by the organizers to produce official results. However, test data have been published after the official evaluation period, and participants could run their systems at home to produce additional results they were allowed to include in the system description papers. There was one main evaluation metric -Labeled Attachment Score -for the main ranking table evaluating dependency parsing performance, plus additional metrics for tokenization, word and sentence segmentation, POS tagging, lemmatization and disambiguation of morphological features, and separate metrics computed for interesting subsets of the evaluation data.A total of 32 systems ran successfully and have been ranked (http://universaldependencies. org/conll17/results.html). While there are clear overall winners, we would like to thank all participants for working hard on their submissions and adapting their systems not only to the datasets available, but also to the evaluation platform. We would like to thank all of them for their effort, since it is the participants who are the core o...

Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP

“…Data To make our analysis maximally comparable across languages, we start from the Parallel Universal Dependencies (PUD) collection (Zeman et al, 2017), which contains translations for a set of 1000 English sentences. PUD only contains test corpora.…”

“…Most of the work on word order variation using Universal Dependencies (UD: de Marneffe et al, 2021) is based on curated dependency treebanks, with only a few works using dependency corpora derived from raw texts. Although the accuracy rate of NLP systems trained on UD models is reportedly very high (Hajič and Zeman, 2017;Zeman and Hajič, 2018;Straka et al, 2019;Qi et al, 2020), a certain level of noise i.e., erroneous annotations is in fact present when working with automatically annotated texts (Levshina et al, to appear; Talamo and Verkerk, to appear); furthermore, different layers of UD annotations such as Universal Parts of Speech (UPOS) and UD Relations are not always used consistently across languages, often resulting in the cross-linguistic comparison of different categories.…”

Tweaking UD Annotations to Investigate the Placement of Determiners, Quantifiers and Numerals in the Noun Phrase

“…Most of the work on word order variation using Universal Dependencies (UD: de Marneffe et al, 2021) is based on curated dependency treebanks, with only a few works using dependency corpora derived from raw texts. Although the accuracy rate of NLP systems trained on UD models is reportedly very high (Hajič and Zeman, 2017;Zeman and Hajič, 2018;Straka et al, 2019;Qi et al, 2020), a certain level of noise i.e., erroneous annotations is in fact present when working with automatically annotated texts (Levshina et al, to appear; Talamo and Verkerk, to appear); furthermore, different layers of UD annotations such as Universal Parts of Speech (UPOS) and UD Relations are not always used consistently across languages, often resulting in the cross-linguistic comparison of different categories.…”

Metrics of Syntactic Equivalence to Assess Translation Difficulty

“…Traditional readability formulas (e.g. Flesch-Kincaid Grade Level (Kincaid et al, 1975), Gunning Fog Index (Gunning, 1952)) typically use shallow source text features such as average word and sentence length and word frequency to assess the reading difficulty level of a given text. Recently, more complex lexical, syntactic, semantic and discourse text features have been used (see for instance Schwarm and Ostendorf (2005); Francois and Miltsakaki (2012);De Clercq et al (2014); De Hoste (2016), andCollins-Thompson (2014) for an overview).…”