No abstract

The main objective of this Methods Showcase Article is to show how the technique of adaptive comparative judgment, coupled with a crowdsourcing approach, can offer practical solutions to reliability issues as well as to address the time and cost difficulties associated with a text-based approach to proficiency assessment in L2 research. We showcased this method by reporting on the methodological framework implemented in the Crowdsourcing Language Assessment Project and by presenting the results of a first study that demonstrated that a crowd is able to assess learner texts with high reliability. We found no effect of language skills or language assessment experience on the assessment task, but judges who had received formal language assessment training seemed to differ in their decisions from judges who had not received such training. However, the scores generated by the crowdsourced task exhibited a strong positive correlation with the rubric-based scores provided with the learner corpus used. Keywords learner corpus; language assessment; proficiency; adaptive comparative judgment; crowdsourcingThe Crowdsourcing Language Assessment Project (CLAP) project was developed within the framework of the Lexicogrammatical Complexity Across Mode T.0086.18 FNRS project. We thank Alex König (then at EURAC, Bolzano, Italy) for his technical help at the start of the project. We are grateful to our colleagues at UCLouvain for taking part in the pilot study and to all national and international colleagues who contributed to CLAP and/or provided feedback on the project. We also thank the reviewers for their very constructive and insightful comments. The usual disclaimers apply. We have no known conflict of interest to disclose.

The extraction of phraseological units operationalized in phraseological complexity measures (Paquot, 2019) relies on automatic dependency annotations, yet the suitability of annotation tools for learner language is often overlooked. In the present article, two Dutch dependency parsers, Alpino (van Noord, 2006) and Frog (van den Bosch et al., 2007), are evaluated for their performance in automatically annotating three types of dependency relations (verb + direct object, adjectival modifier, and adverbial modifier relations) across three proficiency levels of L2 Dutch. These observations then serve as the basis for an investigation into the impact of automatic dependency annotation on phraseological sophistication measures. Results indicate that both learner proficiency and the type of dependency relation function as moderating factors in parser performance. Phraseological complexity measures computed on the basis of both automatic and manual dependency annotations demonstrate moderate to high correlations, reflecting a moderate to low impact of automatic annotation on subsequent analyses.