Linguistic complexity in teachers' assessment of German essays in high stakes testing

Riemenschneider, Anja; Weiß, Zarah; Schröter, Pauline; Meurers, Detmar

doi:10.1016/j.asw.2021.100561

Cited by 6 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We chose this system, because it is to our knowledge the most extensive available analysis system for German. The underlying feature extraction engine for German has proven highly successful and robust in a variety of education-related tasks including readability assessment (Weiss and Meurers, 2018;Kühberger et al, 2019) and work linked to writing quality assessment (Weiss and Meurers, 2019a,b;Weiss et al, 2019;Bertram et al, 2021;Riemenschneider et al, 2021). Also, using a publicly available web-based system increases the re-usability of any model using these features in practice.…”

Section: Spotlight-dementioning

confidence: 99%

See 1 more Smart Citation

Assessing sentence readability for German language learners with broad linguistic modeling or readability formulas: When do linguistic insights make a difference?

Weiß¹,

Meurers²

2022

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

Self Cite

View full text Add to dashboard Cite

The paper presents a new state-of-the-art sentence-wise readability assessment model for German L2 readers. We build a linguistically broadly informed machine learning model and compare its performance against four commonly used readability formulas. To understand when the linguistic insights used to inform our model make a difference for readability assessment and when simple readability formulas suffice, we compare their performance based on two common automatic readability assessment tasks: predictive regression and sentence pair ranking. We find that leveraging linguistic insights yields top performances across tasks, but that for the identification of simplified sentences also readability formulas -which are easier to compute and more accessible -can be sufficiently precise. Linguistically informed modeling, however, is the only viable option for high quality outcomes in finegrained prediction tasks.We then explore the sentence-wise readability profile of leveled texts written for language learners at a beginning, intermediate, and advanced level of German. Our findings highlight that a texts' readability is driven by the maximum rather than the overall readability of sentences. This has direct implications for the adaptation of learning materials and showcases the importance of studying readability also below the document level.

show abstract

Section: Spotlight-dementioning

confidence: 99%

“…The German pipeline used in CTAP is described in detail in and Weiss and Meurers (2021). The latter also contains a comprehensive definition of all complexity measures.…”

Section: Feature Descriptionmentioning

confidence: 99%

Assessing sentence readability for German language learners with broad linguistic modeling or readability formulas: When do linguistic insights make a difference?

Weiß¹,

Meurers²

2022

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

Self Cite

View full text Add to dashboard Cite

show abstract

“…We chose this system, because it is to our knowledge the most extensive available analysis system for German. The underlying feature extraction engine for German has proven highly successful and robust in a variety of education-related tasks including readability assessment (Weiss and Meurers, 2018;Kühberger et al, 2019) and work linked to writing quality assessment (Weiss and Meurers, 2019a,b;Bertram et al, 2021;Riemenschneider et al, 2021). Also, using a publicly available web-based system increases the re-usability of any model using these features in practice.…”

Section: Feature Extraction and Selectionmentioning

confidence: 99%

“…The German pipeline used in CTAP is described in detail in Weiss et al ( 2021) and Weiss and Meurers (2021). The latter also contains a comprehensive definition of all complexity measures.…”

Section: Feature Descriptionmentioning

confidence: 99%

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

2022

View full text Add to dashboard Cite

As scalable learning technologies become ubiquitous, it generates a large amount of student data, which can be used with machine learning and NLP to develop new instructional technologies, such as personalized practice schedules and adaptive lessons. Additionally, machine learning and NLP are uniquely poised to solve the problems inherent in scaling language instruction to a large number of languages and courses. In this talk, I will describe several projects illustrating these two uses of ML and NLP in language learning at scale at Duolingo -the world's largest language education platform with over 100 courses and around 40 million monthly active learners.

show abstract

“…First, based on learner corpora, these features can be used to design metrics exploited for modelling purposes in prediction tasks such as CEFR classification (Vajjala, 2018;Kyle, 2016;Pilán et al, 2016;Yannakoudakis et al, 2011). Secondly, they can be exploited for the design of specific linguistic feedback which is meaningful for learners and teachers (Riemenschneider et al, 2021).…”

Section: Theoretical Backgroundmentioning

confidence: 99%

Language learning analytics : designing and testing new functional complexity measures in L2 writings

Gaillat

2022

Proceedings of the 11th Workshop on Natural Language Processing for Computer-Assisted Language Learning (NLP4CALL 2022)

View full text Add to dashboard Cite

This paper presents the initial stage in the design of an ICALL system. The objective is to develop a system that automatically generates linguistic analytics of L2 learner writings. Student texts will be processed with NLP tools producing different types of textual measures. We present the design of a new functional complexity metric aiming to capture the paradigmatic competition between forms mapped to the same communicative function, i.e. microsystems. More precisely, we analyze the variations of the FOR and TO prepositions in terms of frequency and probability of occurrence. Relative frequency shows significant correlations with CEFR levels suggesting its possible use in an analytics report system. More work is required to extend the approach to other microsystems.

show abstract

Linguistic complexity in teachers' assessment of German essays in high stakes testing

Cited by 6 publications

References 36 publications

Assessing sentence readability for German language learners with broad linguistic modeling or readability formulas: When do linguistic insights make a difference?

Assessing sentence readability for German language learners with broad linguistic modeling or readability formulas: When do linguistic insights make a difference?

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

Language learning analytics : designing and testing new functional complexity measures in L2 writings

Contact Info

Product

Resources

About