Human and Automated CEFR-based Grading of Short Answers

Tack, Anaïs; Thomas, François; Roekhaut, Sophie; Fairon, Cédrick

doi:10.18653/v1/w17-5018

Cited by 10 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tables 1 and 2 give the breakdown for each L1. To test the validity of our models on external data, we used the CEFR ASAG corpus (Tack et al, 2017), a collection of short answers to open-ended questions, written by French L1 learners of English and graded with CEFR levels. It consists of 712 texts written by different learners in response to three questions.…”

Section: Corporamentioning

confidence: 99%

Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach

Gaillat¹,

Simpkin²,

Ballier³

et al. 2021

ReCALL

View full text Add to dashboard Cite

This paper focuses on automatically assessing language proficiency levels according to linguistic complexity in learner English. We implement a supervised learning approach as part of an automatic essay scoring system. The objective is to uncover Common European Framework of Reference for Languages (CEFR) criterial features in writings by learners of English as a foreign language. Our method relies on the concept of microsystems with features related to learner-specific linguistic systems in which several forms operate paradigmatically. Results on internal data show that different microsystems help classify writings from A1 to C2 levels (82% balanced accuracy). Overall results on external data show that a combination of lexical, syntactic, cohesive and accuracy features yields the most efficient classification across several corpora (59.2% balanced accuracy).

show abstract

Section: Corporamentioning

confidence: 99%

Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach

Gaillat¹,

Simpkin²,

Ballier³

et al. 2021

ReCALL

View full text Add to dashboard Cite

show abstract

“…Lexical diversity—a type of lexical richness that refers to the range and variety of words used in a given response—is another aspect of task performance that researchers have looked into as a potential indicator of writing proficiency. According to Tack et al (2017), lexical features—in particular lexical diversity measures—found in short answers of between 30 and 200 words were the most informative predictors (compared with other measures, including syntactic, discursive, and readability features) of English L2 writing proficiency, and they were able to distinguish among A1, A2, B1, B2, and C levels of proficiency on the CEFR. This finding was also confirmed in a study by Crossley et al (2011) in which lexical density ( M ) was found to be one of the best predictors of English L2 writing proficiency among a variety of lexical measures.…”

Section: Study Background and Assessment Contextmentioning

confidence: 99%

Designing Efficient L2 Writing Assessment Tasks for Low‐Proficiency Learners of English

Sasayama¹,

Gomez²,

Norris³

2021

ETS Research Report Series

View full text Add to dashboard Cite

This report describes the development of efficient second language (L2) writing assessment tasks designed specifically for low‐proficiency learners of English to be included in the TOEFL® Essentials™ test. Based on the can‐do descriptors of the Common European Framework of Reference for Languages for the A1 through B1 levels of proficiency, four task types were identified to be prototypical candidate writing tasks for the target test‐taker population (i.e., adolescent and adult low‐proficiency English learners). Those four task types included: (a) Describe a Photo, (b) Write a Review, (c) Chat With a Friend, and (d) Write an E‐mail. These task types were also considered efficient in the framework of the test in that they had the potential to be accessible to low‐proficiency learners and to elicit sufficient spontaneous writing for assessment purposes within a short period of time. In the current study, eight assessment tasks, two for each task type, were developed and piloted with 169 A1–B1 learners of English from Japan and Colombia. The findings revealed that the Describe a Photo and Write an E‐mail tasks performed the best in eliciting substantial language use and emphasizing distinct performance attributes, both characteristics needed for efficiently measuring test takers' writing proficiency as well as discriminating among proficiency levels at the lower end of the spectrum. The report concludes by highlighting some observations on L2 writing assessment task design for low‐proficiency learners of English.

show abstract

“…The problem of SAQ scoring has received considerable attention, with several shared tasks and competitions orgnaized in the past (e.g., a SemEval shared task (Dzikovska et al, 2013), or the ASAP 2 Kaggle competition 3 ). The task is to predict the human labels for each instance and, traditionally, this has been done using n-grams (Heilman and Madnani, 2015) or a wide variety of linguistic features as in Tack et al (2017). Leacock and Chodorow (2003) use predicate-argument structure, pronominal reference, morphological analysis and synonyms to rate the questions.…”

Section: Automated Saq Scoringmentioning

confidence: 99%

Automated Prediction of Examinee Proficiency from Short-Answer Questions

Yaneva

Harik

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

This paper brings together approaches from the fields of NLP and psychometric measurement to address the problem of predicting examinee proficiency from responses to short-answer questions (SAQs). While previous approaches train on manually labeled data to predict the human ratings assigned to SAQ responses, the approach presented here models examinee proficiency directly and does not require manually labeled data to train on. We use data from a large medical exam where experimental SAQ items are embedded alongside 106 scored multiple-choice questions (MCQs). First, the latent trait of examinee proficiency is measured using the scored MCQs and then a model is trained on the experimental SAQ responses as input, aiming to predict proficiency as its target variable. The predicted value is then used as a "score" for the SAQ response and evaluated in terms of its contribution to the precision of proficiency estimation.

show abstract

Human and Automated CEFR-based Grading of Short Answers

Cited by 10 publications

References 16 publications

Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach

Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach

Designing Efficient L2 Writing Assessment Tasks for Low‐Proficiency Learners of English

Automated Prediction of Examinee Proficiency from Short-Answer Questions

Contact Info

Product

Resources

About