Applying Content Similarity Metrics to Corpus Data: Differences Between Native and Non‐native Speaker Responses to a Toefl® Integrated Writing Prompt

Abstract

For many purposes, it is useful to collect a corpus of texts all produced to the same stimulus, whether to measure performance (as on a test) or to test hypotheses about population differences. This paper examines several methods for measuring similarities in phrasing and content and demonstrates that these methods can be used to identify population differences between native and non‐native speakers of English in a writing task.

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

“…Finally, applying detection methods to practical applications in high‐stakes settings is nontrivial. Information from other modalities should also be considered, such as the writing process information (Deane, et al,2018; Choi et al,2021; Hao & Fauss, 2023). As AI tools and human adaptability continue to evolve, detection methods should constantly improve to stay relevant and reliable.…”

Performance Assessment

“…C-rater is an automated scoring method for scoring constructed-response items that elicit verbal responses that range from one sentence to a few paragraphs, have rubrics that explicitly specify the content required in the response, but do not evaluate the mechanics of writing items (Leacock & Chodorow, 2003. It has been used successfully in Indiana' s state end-of-course, grade 11 English assessment, the NAEP Math Online project that required students to provide explanations of their mathematical reasoning, and the NAEP simulation study that required students to use search queries (Bennett et al, 2007;Deane, 2006). C-rater is a paraphrase recognizer in that it can determine when a student' s constructed response matches phrases in the scoring rubric regardless of their similarity in word use or grammatical structure (Deane, 2006).…”

“…It has been used successfully in Indiana' s state end-of-course, grade 11 English assessment, the NAEP Math Online project that required students to provide explanations of their mathematical reasoning, and the NAEP simulation study that required students to use search queries (Bennett et al, 2007;Deane, 2006). C-rater is a paraphrase recognizer in that it can determine when a student' s constructed response matches phrases in the scoring rubric regardless of their similarity in word use or grammatical structure (Deane, 2006). In the NAEP study that used physics computer-based simulations, c-rater models were built using student queries and then cross-validat-Performance Assessment: The State of the Art ed using a sample of queries that were independently hand-scored.…”