Assessing the impact of automatic dependency annotation on the measurement of phraseological complexity in L2 Dutch

Abstract

The extraction of phraseological units operationalized in phraseological complexity measures (Paquot, 2019) relies on automatic dependency annotations, yet the suitability of annotation tools for learner language is often overlooked. In the present article, two Dutch dependency parsers, Alpino (van Noord, 2006) and Frog (van den Bosch et al., 2007), are evaluated for their performance in automatically annotating three types of dependency relations (verb + direct object, adjectival modifier, and adverbial modifier relations) across three proficiency levels of L2 Dutch. These observations then serve as the basis for an investigation into the impact of automatic dependency annotation on phraseological sophistication measures. Results indicate that both learner proficiency and the type of dependency relation function as moderating factors in parser performance. Phraseological complexity measures computed on the basis of both automatic and manual dependency annotations demonstrate moderate to high correlations, reflecting a moderate to low impact of automatic annotation on subsequent analyses.

Proceedings of the 12th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2023)

“…We will first perform the annotation of verbal tenses in both corpora using our script. However, since automatic language processing tools are developed on the basis of well-formed data, it is to be expected that learner corpora, due to the inclusion of errors, will lead to annotation errors (Granger, 2011;Štindlová et al, 2011;Krivanek and Meurers, 2013;Rubin, 2021;. Therefore, we decided to also manually annotate the learner corpus.…”

A Dependency Treebank of Spoken Second Language English

“…For more precise insights into linguistic development, some studies have constrained the lexical combinations that are used (e.g., adjective + noun or noun + noun combinations). Even more recently, researchers have begun to use dependency parses to analyze lexical items in particular grammatical relationships (e.g., verb + object; Kyle & Eguchi, 2021;Paquot, 2018Paquot, , 2019Rubin, 2021). Lexicogrammatical features: A number of studies have investigated the relationship between L2 proficiency and the use of lexicogrammatical features that are common in academic writing such as various types of noun phrase elaboration (e.g., Biber et al, 2014;Grant & Ginther, 2000;Picoral et al, 2021).…”

“…evaluated the accuracy of verb argument construction identification using a sample of 100 sentences from a corpus of L2 essays. Similar procedures have been used in a number of other studies (e.g., Díez-Bedmar & Pérez-Paredes, 2020;Paquot, 2019;Rubin, 2021). While small-scale accuracy analyses are important for establishing the effectiveness of particular linguistic analysis tools for a particular data set, these datasets are rarely made publicly available and do not necessarily follow the annotation guidelines or formatting conventions of well-known treebanks.…”

Towards a Verb Profile: distribution of verbal tenses in FFL textbooks and in learner productions

“…We will first perform the annotation of verbal tenses in both corpora using our script. However, since automatic language processing tools are developed on the basis of well-formed data, it is to be expected that learner corpora, due to the inclusion of errors, will lead to annotation errors (Granger, 2011;Štindlová et al, 2011;Krivanek and Meurers, 2013;Rubin, 2021;Volodina et al, 2022). Therefore, we decided to also manually annotate the learner corpus.…”